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1.0  INTRODUCTION 


This  volume  applies  the  matching  principles  and  tax- 
onomies introduced  in  Volume  I and  detailed  in  Volume  111  to 
five  actual  decision  situations.  The  purpose  of  this  volume 
is  three-fold: 

1.  To  illustrate  the  idea  of  matching  principles. 

2.  To  uncover  weaknesses  in  the  existing  developments 
of  the  taxonomies  and  suggest  ways  of  correcting  these 
weaknesses. 


3.  To  test  the  viability  of  the  taxonomies  as  a 
framework  for: 

a)  Communicating  matching  principles  to  decision 
makers  and  decision  analysts  and 

b)  Developing  matching  principles  that  identify 
the  proper  analytic  techniques  for  decision  situations. 

Throughout  this  volume,  our  explanations  of  the  reasons 
surrounding  the  choice  of  analytic  techniques  are  stated  in 
terms  of  the  taxonomies,  but  the  applications  of  matching 
principles  are  much  more  informal  than  those  presented  in 
the  Export  Control  Decision  (Section  4 of  Volume  I) . 

The  five  cases  presented  in  this  volume  are: 

Case  1 — A Foreign  Policy  Decision 

Case  2 — A Business  Investment  Decision 

Case  3 — A Design-to-Cost  Evaluation 
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Case  4 — A Navy  Task  Force  Conunander's  Tactical  Decision 


Case  5 — Predicting  NATO's  Response. 

These  cases  were  chosen  to  represent  a broad  range  of  situations 
in  which  decision  analysis  has  been  used  successfully.  The 
first  case  refers  to  the  problem  of  what  foreign  policy 
position  the  US  should  adopt  in  order  to  obtain  more  oil 
from  a particular  mideastern  country.  The  second  case 
refers  to  a decision  faced  by  the  president  of  an  electrical 
equipment  company  of  whether  to  purchase  manufacturing 
rights  to  a flight  safety  patent.  The  third  case  refers  to 
a project  undertaken  by  the  Naval  Electronics  Systems  Command 
to  apply  the  "Design- to-Cost"  concept  to  an  evaluation  of 
proposed  electronic  warfare  systems.  The  fourth  case  refers 
to  a research  and  development  project  aimed  at  developing 
tactical  decision  aids  for  Navy  task  force  commanders.  The 
fifth  case  refers  to  a study  aimed  at  predicting  NATO's 
response  to  actions  by  Warsaw  Pact  countries.  We  recognize 
that  this  choice  of  cases  causes  our  presentation  to  be 
weighted  heavily  toward  decisions  that  favor  the  use  of  at 
least  some  amount  of  decision  analysis.  However,  decision 
analysis  was  generally  justified  by  different  combinations 
of  reasons  from  case  to  case,  and  different  specific  analytic 
techniques  were  used  in  different  cases. 

The  length  of  descriptions  also  varies  with  the  cases. 

The  first  three  presentations  are  longer  than  the  last  two. 

The  lengths  are  indicative  of  both  the  number  of  important 
analytic  techniques  used  in  the  cases  and  the  extent  to 
which  the  taxonomies  were  found  to  be  useful  in  explaining 
the  analytic  choices.  In  most  of  the  cases,  we  address  our 
attention  to  only  the  most  important  analysis  choices. 

However,  in  the  second  case,  we  present  a more  complete  and 
extensive  application  of  the  taxonomies. 
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Although  the  applications  of  the  matching  principles  to 
the  cases  were  performed  after  the  fact,  we  have  generally 
attempted  to  reconstruct  the  events  of  the  cases  to  illus- 
trate how  the  matching  principles  might  have  been  applied  in 
the  unfolding  situation. 

To  obtain  a complete  understanding  of  the  arguments 
presented  in  this  volume,  the  reader  may  need  to  refer  to 
the  detailed  descriptions  of  the  taxonomies  presented  in 
Volume  III.  However,  we  consider  such  references  unnecessary 
in  a first  reading.  In  addition,  the  reader  may  wish  to 
refer  to  more  technical  descriptions  of  the  case  studies 
which  are  contained  in  other  technical  reports.  Requests 
for  copies  of  these  reports  should  be  addressed  to: 

Librarian 

Decisions  and  Designs,  Inc. 

Suite  600 

8400  Westpark  Drive 

McLean,  Virginia  22101 

Throughout  this  volume,  references  to  the  situation 
taxonomy  of  Section  A,  Volume  III  are  prefixed  with  the 
letter  "S,"  references  to  the  analytic  taxonomy  of  Section 
B,  Volume  III  are  prefixed  with  the  letter  "A,"  and  references 
to  the  performance  measure  taxonomy  of  Section  C,  Volume  III 
are  prefixed  with  the  letter  "P." 


0 CASE  1:  A FOREIGN  POLICY  DECISION 


2.1  Introduction 


This  case  refers  to  a decision  analysis  of  alternative 
foreign  policy  positions  that  the  United  States  could  adopt 
in  order  to  obtain  more  oil  from  a particular  Mideastern 
country.  A technical  description  of  this  analysis  appears 
in  brown,  Peterson,  and  Ulvila  (1975)  and  is  not  reproduced 

here. 


2.1.1  Problem  background  - In  the  early  1970' s,  before 
the  "energy  crisis"  of  October  1973,  U.S.  government  policy 
makers  had  begu  ' to  recognize  the  potential  for  such  a 
crisis.  U.S.  energy  needs  were  expected  to  climb  steadily 
over  the  next  decade.  However,  the  supply  of  energy,  which 
was  hard-pressed  to  meet  the  current  demand,  was  not  expected 
to  expand  during  this  time.  Thus,  the  prospects  for  the 
near-term  future  were  that  the  supply  of  energy  would  fall 
seriously  short  of  demand. 

In  response  to  these  prospects,  a top-level  U.S. 
government  policy  maker  (referenced  as  the  policy  maker  in 
the  future)  established  an  inter-agency  sub-committee  com- 
posed of  representatives  from  the  Conurierce  Department,  the 
Interior  Department,  the  State  Department,  the  Defense 
Department,  and  others.  This  sub-committee  was  charged  with 
the  responsibility  of  making  a recommendation  of  a foreign 
policy  position  that  would  improve  the  supply  of  oil  to  the 
U.S.  The  chairman  of  the  committee  (referenced  as  the 
chairman  in  the  future) , a government  policy  advisor,  was 
ultimately  responsible  for  making  the  recommendation  to  the 
policy  maker. 
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As  the  conunittee  began  to  investigate  the  problem, 
its  chairman  felt  that  the  deliberations  were  especially 
disconnected,  haphazard,  vague,  and  unproductive.  This 
situation  was  especially  discomforting  to  the  chairman 
because  he  knew  that  his  boss,  the  policy  maker,  demanded 
carefully  reasoned  and  logically  consistent  policy  recommen- 
dations. Thus  the  chairman  was  convinced  that  his  recommen- 
dation would  not  be  acted  upon  unless  it  was  based  upon 
careful  and  consistent  reasoning. 

The  chairman  was  familiar  with  the  efforts  of 
Decisions  and  Designs,  Inc.  (DDI)  in  applying  decision 
analysis  to  government  policy  questions.  He  felt  that  their 
methodology  might  help  him  sort  through  the  vast  amount  of 
information  that  he  had  obtained  about  his  problem  from 
various  experts  and  help  him  to  build  a logically  sound 
recommendation.  With  this  in  mind,  the  chairman  contacted 
DDI  to  initiate  a decision  analysis  of  this  problem. 

2.1.2  Use  of  decision  analysis  - This  project  repre- 
sented an  initial  attempt  of  the  chairman  to  apply  decision 
analysis.  As  such,  this  analysis  was  viewed  as  an  opportunity 
to  experiment  with  a new  approach  to  policy  analysis.  If 
this  experiment  proved  successful,  it  was  likely  that  the 
chairman  would  use  decision  analysis  for  future  policy 
analyses. 


The  first  task  in  the  analysis  was  to  define  the 
decision  options.  Within  the  main  problem  area  of  improving 
the  oil  position  of  the  U.S.,  the  committee  saw  an  agreement 
with  a Mideastern  oil-producing  country  as  the  most  promising 
region  for  investigation.  However,  even  within  this  narrowly 
defined  region,  the  possible  range  of  agreements  was  enormous, 
and  their  effects  would  be  wide  and  far  reaching. 
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The  analysis  eventually  settled  down  to  an 
investigation  of  unilateral  concessions  that  the  U.S.  could 
make  to  a particular  mideastern  country.  The  chairman  felt 
that  the  U.S.  could  offer  the  most  to  this  country  along 
four  main  dimensions;  political  support,  military  support, 
dollar  absorption,  and  cultural  exchange.  Within  the  context 
of  these  dimensions,  three  representative  concessions  were 
structured  for  future  evaluation  in  the  analysis. 

Although  these  three  concessions  would  impact  on 
a wide  range  of  U.S.  interests,  the  chairman  was  willing  to 
evaluate  them  on  the  basis  of  their  impacts  on  five  key  U.S. 
interests.  The  first  interest  was  the  effect  that  the 
concession  would  have  on  the  volume  and  price  of  oil  that 
the  mideastern  country  would  offer  to  the  U.S.  The  second 
interest  was  the  effect  of  the  concession  on  the  United 
State's  balance  of  payments.  Third  was  the  effect  on  the 
relationship  between  the  U.S.  and  its  allies.  Fourth  were 
the  sentiments  of  the  U.S.  public  to  the  concession.  Fifth 
was  the  effect  that  the  concession  would  have  on  the  oil 
that  other  countries  would  make  available  to  the  U.S. 

It  was  readily  apparent  to  both  the  chairman  and 
the  analysts  that  the  effects  of  the  concessions  on  the 
valuation  criteria  were  very  uncertain.  Furthermore,  the 
valuation  criteria  themselves  were  imprecisely  defined.  In 
particular,  they  could  be  subdivided  very  easily  into  many 
component  parts. 

The  role  of  the  decision  analysis  was,  then,  to 
generate  decision  options  that  could  be  analyzed,  structure 
a framework  for  the  analysis,  and  use  the  judgments  of 
various  experts  to  arrive  at  a policy  recommendation. 
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2 . 2 Performance  Determinants  of  a '^ood  Analysis 


The  important  situation  characteristics  of  this  case 
can  be  distilled  in  the  form  of  performance  measures.  This 
particular  case  required  three  major  types  of  performance. 

The  analysis  had  to  process  inputs  effectively,  reach  a 
logically  sound  decision,  and  communicate  its  reasor.ing.  In 
addition,  the  cost  of  the  analysis  could  not  overrun  the 
budget. 

A large  quantity  of  data  was  available  at  the  beginning 
of  the  case.  Because  this  data  was  vital  to  the  decision, 
the  most  important  measure  of  the  quality  of  the  analysis 
was  its  ability  to  process  the  existina  data  inputs  (P122).^ 

In  addition,  the  situation  was  characterised  by  options, 
values,  and  uncertainties  that  were  all  difficult  to  define 
and  assess.  Thus  another  comoonent  of  input  processing  was 
the  ability  of  the  analysis  to  pose  meaningful  questions 
about  options,  values,  and  uncertainties  to  the  assessors 
(P123) . 

Because  the  stakes  involved  in  the  analysis  were  high 
(S1444)  and  the  best  decision  was  not  obvious  (S1334) , the 
quality  of  the  reasoning  and  logic  contained  in  the  analysis 
(P144)  was  the  second  most  important  measure  of  a good 
analysis.  Three  components  of  logic  were  most  important. 

In  descending  order  of  importance  the  components  were, 
first,  the  analysis  had  to  provide  an  effective  disaggregation 
of  the  problem  (P112) ; it  had  to  divide  the  problem  into  manage 
able  sub-problems.  Second,  the  analysis  had  to  address  the 

^References  that  begin  with  the  letter  "P"  are  performance 
measures,  which  are  explained  in  detail  in  Appendix  C of 
Brown  and  Ulvila  (1977) . 

^References  that  begin  with  the  letter  "S"  are  situation 
characteristics,  which  are  explained  in  Section  2.5  below. 
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entire  problem  (P115) , not  just  a part  of  the  problem  such 
as  the  assessment  of  probabilities.  Third,  the  analysis  had 
to  be  conceptually  complete  (Pill) ; it  had  to  account  for 
all  important  considerations  and  avoid  serious  approximation 
errors. 

The  third  most  important  measure  of  a good  analysis  was 
its  ability  to  communicate  the  reasons  for  a decision  (P321) . 
This  performance  measure  was  especially  important  in  this 
case  because  the  decision  maker  would  have  to  justify  his 
decision  to  his  boss  and  to  others. 

2. 3 Analytic  Choices 

The  recommended  analytic  options  for  this  case  are 
summarized  on  Table  2-1.  The  following  sections  explain  the 
reasoning  behind  the  most  important  choices. 

2.3.1  Choice  of  the  amount  of  analysis  - Section  3.1.2 
of  Volume  I contains  an  explanation  of  the  guidelines  for 
determining  the  proper  amount  of  decision  analysis  in  terms 
of  situational  characteristics.  The  decision  maker  could 
have  characterized  his  situation  along  these  dimensions  to 
determine  the  proper  amount  of  decision  analysis  as  follows. 

Important  factors  favoring  a large  amount  of 
decision  analysis; 

o The  choice  of  the  best  foreign  policy  recommenda- 
tion was  difficult  (S1313) . 

o The  choice  involved  high  stakes.  The  expected 

difference  in  value  among  the  reasonable  options 
was  on  the  order  of  several  hundred  million  dollars 
of  the  U.S.  federal  budget  (S1444) . 
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USER'S  OPTIONS 


3  INPUT  STRUCTURE 


11  USE  DECISION  ANALYSIS  AT  ALL? 

11"?  vei 

12  DOLLAR  AMOUNT  OF  ANALYSIS 

122  medium 

13  ROLE  OF  DECISION  ANALYSIS 

1312  public 
1322  optional 
1342  display 

14  ORGANIZIATION 

1419  consultants  to  perform  the  analysis 
14218i3  decision  makers  and  experts  to 
provide  inputs 

1421  close  relationship  with  decision 
maker 

16  RESOURCES 

1511  decision  analysis  specialists 
1521  computer 

MODEL  APPROACH  OPTIONS 

21  complexity  OF  ANALYSIS 

212  moderately  complex 

22  COMPREHENSIVE  OR  PARTIAL 
2223  partial  on  options 

23  DEGREE  OF  APPROXIMATION 

232  medium  exactness 

24  BALANCE  OF  EFFORT 

2412  appreciable  effort  on  option 
definition 

24212  moderate  effort  on  uncertainty 
modeling 

24222  moderate  effort  on  value  modeling 


31  UNCERTAINTY 

3110  certainty  equivalents 
3121  short  time  horizon 

3130  no  subsequent  acts 

3141  low  detail 
3162  high  degree  of  grouping 

32  VALUE 

3213  comprehensive 
32312  single  utility  index 
32411  linear  value  function 

4 INPUT  SPECIFICATION 

43  VALUE  CRITERIA 

4319  mixture  of  units 
4322  "floating-zero"  base 

45  ELICITATION  TECHNIQUE 

4512  direct  for  probabilities 

4532  direct  for  values 

5 OUTPUT 

51  SPECIFICATION 

5121  single  value  for  each  option 

52  DISPLAY  FORMAT 
5211  graphic 


Table  2-1 

SUMMARY  OF  THE  ANALYSIS 
CHOICES  FOR  THE  FOREIGN  POLICY  DECISION 
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o Many  sources  of  inputs  were  vital  to  the  decision 
(S2215)  . 

o The  decision  maker  was  required  to  justify  his 
decision  to  his  boss  and  to  others  (S2343). 

o The  decision  maker  was  familiar  with  decision 
analysis  (S24212)  . 

Important  factors  against  a large  decision 

analysis : 

o The  available  decision  options  were  not  clearly 
defined  (S1222) . 

o Option  generation/  rather  than  choice/  was  the  key 
determinant  of  a good  decision  (S1331) . 

Other  factors  favoring  a J.arge  decision  analysis: 

o The  choice  was  current ; the  chairman  would  make  a 
recommendation  in  60  days  (Sllll) . 

o The  chairman  expected  to  make  very  similar  deci- 
sions in  the  future  (S1124). 

o The  outcomes  were  difficult  to  evaluate  on  the 
value  dimensions  (S1512) . 

o The  uncertain  quantities  were  difficult  to  assess 
(S1621) . 

o The  decision  maker  had  a relatively  long  time/  60 
dayS/  to  make  his  recommendation  CS214) . 
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o This  analysis  was  considered  a training  project 
and  it  might  determine  whether  decision  analysis 
would  be  used  in  the  future  (S2252) . 

o The  "rational  actor"  model,  which  hypothesizes  a 
single  decision  maker  who  relies  on  reasoned 
judgments  in  making  his  decisions,  fits  this  case 
very  well  (S2372) . 

o Computer  facilities  were  readily  available  (S2512) . 

o A strong  staff  was  available  (S2522) . 

o The  decision  maker  was  available  for  frequent 
interactions  with  the  analysts  (S2542). 

Another  factor  against  a large  decision  analysis; 

o Only  a modest  budget  was  available  for  the  analysis 
(S2562) . 

In  this  case,  many  more  factors  argued  for  using 
a large  amount  of  decision  analysis  than  argued  against  it. 
Most  importantly,  the  stakes  in  the  decision  were  high,  the 
choice  was  fairly  difficult,  and  enough  time  was  available 
to  perform  a large  analysis.  Thus,  the  decision  situation 
indicated  that  a large  decision  analysis  was  needed.  How- 
ever, only  a modest  budget  was  available  for  the  analysis, 
so  a large  analysis  was  impossible.  Therefore,  the  size  of 
the  analysis  was  ultimately  decided  by  the  size  of  the 
analysis  budget. 

2.3.2  Type  of  analysis  - The  following  ten  analytic 
choices  were  made  mainly  by  the  decision  analysts  during  the 
course  of  the  analysis  effort. 
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The  first  major  analytic  choice  was  whose  point 
of  view  the  model  should  take.  In  making  his  recommendation, 
the  decision  maker  was  motivated  to  consider  what  was  best 
for  the  United  States,  rather  than  what  was  best  for  himself 
personally.  Thus,  the  analysis  should  have  taken  the  role 
of  a public  analysis  (A1312) , one  that  is  performed  for  the 
benefit  of  an  institution  rather  than  an  individual.  This 
was  done  in  the  actual  analysis. 

The  second  major  analytic  choice  was  to  build  a 
partial  model  of  the  decision  options  (A2223)  initially  and 
to  complete  this  part  of  the  model  as  dictated  by  a sensitivity 
analysis.  In  this  case,  the  possible  decision  options  were 
very  numerous  and  complex  (S1236)  so  it  was  important  to  try 
to  develop  the  analysis  on  a few  fairly  obvious  options 
before  attempting  to  evaluate  other  options.  This  strategy 
was  particularly  appropriate  in  this  case  because  it  was 
likely  that  an  analysis  developed  on  a few  obvious 
decision  options  would  apply  to  other  options.  That  is,  the 
structure  appropriate  for  analyzing  a few  options  would  also 
be  appropriate  for  analyzing  most  of  the  other  options. 

The  reader  should  notice  that  the  reasoning 
surrounding  this  analytic  choice  does  not  fit  conveniently 
into  the  framework  of  the  taxonomies  as  they  are  currently 
developed.  This  is  because  it  is  impossible  to  describe 
decision  situations  in  a way  that  is  universally  applicable 
to  all  analytic  choices  without  using  a much  finer  level  of 
detail  than  is  currently  developed  in  the  taxonomies.  At 
the  present  level  of  development,  the  right  way  to  describe 
a situation  for  one  choice  may  differ  from  the  right  way  to 
describe  a situation  for  another  choice.  This  problem  can 
be  overcome  by  developing  the  existing  universal  taxonomies 
to  a finer  level  of  detail  or  by  developing  taxonomies  that 
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are  specific  to  particular  fields  of  decision  making.  (The 
idea  of  specific  taxonomies  is  explained  in  greater  detail 
in  Section  4.2.2  of  Volume  I.)  However,  such  a development 
would  be  a vary  major  undertaking  and  is  outside  of  the 
scope  of  the  present  project. 

The  third  major  analytic  choice  was  to  use  the 
analysis  in  a display  mode  (A1342) . That  is,  the  choice  was 
made  to  use  the  analysis  to  display  the  considerations 
surrounding  the  decision  rather  than  to  show  only  the  optimum 
decision.  We  would  almost  always  recommend  some  amount  of 
display,  but  in  this  case,  because  the  decision  needed  to  be 
justified  (S2343)  and  the  decision  options  were  modeled 
partially  (A2223) , the  display  mode  was  especially  important. 

The  fourth  major  analytic  choice  concerned  the 
balance  of  effort  among  the  various  parts  of  the  analysis. 

In  the  actual  analysis,  approximately  90%  of  the  effort  was 
spent  on  developing  several  detailed  models  of  the  oil  that 
the  Mideastern  country  would  make  available  to  the  U.S.  In 
retrospect,  the  balance  of  effort  should  have  been  spread 
more  evenly  across  the  various  analysis  areas.  In  particular, 
because  the  value  judgments  were  controversial  (S2352) , a 
much  larger  portion  of  the  effort  should  have  been  allocated 
to  making  the  tradeoffs  among  the  value  dimensions  (A24222) . 
Also,  because  component  valuation  was  difficult  (S1563)  , 
slightly  more  of  the  remaining  (reduced)  amount  of  effort  to 
be  devoted  to  modeling  the  uncertain  outcomes  (A24212) 
should  have  been  devoted  to  modeling  the  other  outcomes  of 
allied  goodwill,  U.S.  sentiments,  and  balance  of  payments. 

The  allied  goodwill  dimension  should  have  been  decomposed 
into  the  effect  of  the  decision  options  on  each  of  several 
allies,  such  as  Germany,  France,  and  Japan.  The  U.S.  sentiment 
dimension  should  have  considered  different  segments  of  the 
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U.S.  population.  The  balance  of  payments  dimension  should 
have  been  decomposed  into  elements  such  as  export  business, 
import  business,  oil  imports,  and  so  forth.  The  overall 
effects  of  these  recommendations  would  be  to  greatly  reduce 
the  percentage  of  analytic  effort  spent  on  the  models  of 
oil.  The  taxonomic  framework  may  have  helped  the  analysts 
in  this  case  to  avoid  this  problem  by  bringing  the  question 
of  the  proper  balance  of  effort  to  their  attention.  Such 
prompting  may  have  corrected  their  tendency  to  prematurely 
limit  the  analysts  by  modeling  what  was  most  immediately 
accessible  to  them.^ 

The  fifth  major  analytic  choice  was  to  convert 
all  outcome  valuations  to  a single  index,  dollars  in  the 
U.S.  federal  budget  (S32312).  This  choice  was  indicated 
because  the  evaluations  of  the  outcomes  along  the  five 
dimensions  of  value  did  not  combine  naturally  (SI 541) . In 
particular,  values  were  expressed  in; 

o Barrels  per  day  and  dollars  on  the  value  dimension 
of  oil  from  the  Mideastern  country. 

o Dollars  on  the  value  dimension  of  balance  of 
payments . 

o Abstract  utility  units  on  the  three  value  dimensions 
of  allied  goodwill,  U.S.  sentiments,  and  other 
sources  of  oil. 


Another  example  of  a case  where  the  analysts  yielded  to 
temptation  to  model  the  most  immediately  accurate  part  of 
the  problem  is  documented  in  the  Allied  case  (pp.  520-547  of 
Brown,  Kahr,  and  Peterson  [1974]).  In  that  case,  the  analysts 
decided  immediately  to  model  the  value  of  Allied's  shock 
absorber  business  rather  than  the  more  appropriate  (by  hind- 
sight) decision  of  what  they  could  do  if  they  left  the  shock 
absorber  business. 


The  atti activeness  of  the  decision  options  would  be  impossible 
to  determine  by  trading  off  these  value  units  directly. 

Thus,  everything  was  converted  into  a single  measure.  In 
addition,  the  particular  measure,  federal  budget  dollars, 
was  chosen  because  utility  was  assessed  as  linear  with 
federal  budget  dollars;  and  this  measure  was  one  that  the 
policy  maker,  who  would  actually  act  on  the  recommendation, 
could  easily  understand. 

Two  other  methods  of  arriving  at  a single  valua- 
tion measure  were  considered,  but  these  were  not  as  attractive 
as  converting  all  of  the  values  into  federal  budget  dollars. 
First,  all  of  the  values  could  have  been  converted  into  an 
abstract  utility  figure,  but  the  meaning  of  the  figure  would 
have  been  very  difficult  to  communicate  to  the  policy  maker 
(who  was  unfamiliar  with  decision  analysis) . Second,  all  of 
the  values  could  have  been  converted  into  the  units  of  one 
of  the  components  (e.g.  barrels  per  day  of  oil),  This  was 
not  done  because  utility  was  non-linear  in  each  component's 
value  dimension.  Thus,  assessments  would  be  much  more 
difficult  if  a component's  value  measure  was  used  instead  of 
the  linear  measure  of  federal  budget  dollars. 

The  sixth  analytic  choice  was  to  use  three 
parallel  approaches  to  modeling  the  uncertainty,  "oil  from 
the  Mideastern  country."  Using  several  approaches  for  the 
same  uncertainty  improved  the  logic  of  the  analysis  (Pll)  by 
providing  several  checks  on  the  accuracy  of  the  assessment. 

In  this  case,  three  models  of  the  same  quantity  yielded 
similar  outputs.  A further  argument  for  this  approach  is 
that  the  uncertain  quantity  lent  itself  naturally  to  the  use 
of  several  modeling  techniques.  Thus,  the  parallel  models 
could  be  built  relatively  quickly  and  easily. 


15 


In  particular,  three  approaches  were  used: 
direct  assessment,  conditioned  assessment,  and  joint  assess- 
ment. In  the  direct  assessment,  the  assessor  specified  a 
probability  distribution  of  the  volume  of  oil  that  the  Mid- 
eastern country  would  make  available  to  the  U.S.  under  each 
of  the  three  decision  options.  The  direct  method  was  used 
because  it  is  generally  the  easiest  method  to  use  if  the 
assessors  are  willing  to  express  their  judgments  directly, 
as  they  were  in  this  case.  The  conditioned  assessment  of 
oil  volume  was  indicated  because  this  was  the  most  natural 
way  for  the  assessors  tr  ' hink  about  oil  volume.  That  is, 
even  though  the  assessors  were  willing  to  make  direct  judg- 
ments on  the  volume  of  oil,  their  first  reaction  to  questions 
about  this  volume  was,  "It  all  depends."  A conditioned 
assessment  model  explicitly  treats  the  dependencies  and  thus 
was  a natural  model  to  use  in  this  case.  The  joint  distribu- 
tion model,  which  called  for  an  assessment  of  the  joint 
probability  distribution  of  the  price  of  oil  and  the  volume 
of  oil,  was  used  because  the  decision  maker  was  predisposed 
to  evaluate  the  attractiveness  of  the  concessions  based  on 
both  the  price  and  volume  of  oil  that  would  result.  Thus, 
the  joint  assessment  was  chosen  because  it  made  valuation 
easier.  This  method,  however,  made  the  assessment  of  the 
uncertainty  very  difficult.  In  particular,  the  method 
required  the  analysts  to  develop  a new  analytic  method  for 
eliciting  the  joint  distribution.  In  retrospect,  this 
approach  should  not  have  been  used,  since  it  was  too  elaborate 
and  sophisticated  to  be  communicated  effectively.  In  addition, 
using  this  approach  contributed  to  the  disproportionate 
eunount  of  effort  spent  on  the  model  of  oil  from  the  Mid- 
eastern country. 

The  seventh  main  analytic  choice  was  to  obtain 
inputs  for  the  model  using  informal  group  elicitation  tech- 
niques (A45411) . This  choice  was  indicated  because  the 
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views  of  many  experts  needed  to  be  considered  in  the  analysis 
(S2215) , and  group  elicitation  methods  are  the  cheapest  ways 
to  extract  the  opinions  of  all  of  the  experts.  The  informal 
method  was  indicated!  rather  than,  for  instance,  the  Delphi 
method,  because  each  expert's  status  was  the  Scime,  but  each 
was  an  authority  in  a different  subject  area  (e.g.,  politics, 
economics,  and  the  oil  industry) . 

The  eighth  major  analytic  choice  was  to  use  the 
method  of  "swing  weights"  to  evaluate  the  outcomes  of  the 
decision  options  and  aggregate  those  values  into  a single 
figure.  This  method  of  swing  weights  combines  three  analytic 
techniques.  First,  a "floating  zero"  base  (A4322)  was  used 
for  the  outcome  valuations.  On  each  value  dimension,  the 
value  of  each  decision  option  was  measured  relative  to  the 
value  of  the  "baseline"  of  "no  change  in  policy."  Second, 
the  units  of  value  were  defined  by  specifying,  on  each 
value  dimension,  the  "swing"  in  value  between  best  decision 
option  and  worst  decision  option  as  100  units  (A4319) . 

Third,  the  tradeoffs  among  value  swings  on  the  dimensions 
were  specified  as  "weights,"  thus  forming  a linear  value 
function  (A3241i) . 

This  method  was  chosen  because  it  promoted  a 
compact  elicitation  of  values  that  was  easy  to  communicate 
to  the  decision  maker.  This  method  is  compact  because  it 
only  poses  questions  that  are  directly  relevant  to  the 
problems  being  analyzed.  That  is,  the  elicitations  that  are 
used  to  establish  the  value  scales  also  serve  to  evaluate 
the  decision  options  on  those  scales.  Other  forms  of  value 
models  may  require  separate  elicitations  just  to  establish 
the  value  scale.  In  addition,  the  value  scales  are  easy  to 
communicate  because  they  are  directly  tied  to  the  options 
being  evaluated. 
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These  features  make  the  swing-weight  method  an 
efficient  one  for  analyzing  a particular  problem,  but  they 
also  tie  the  value  scale  to  the  analysis  in  such  a way  that 
it  becomes  very  difficult  to  use  for  other  problems.  That 
is,  a swing-weight  value  scale  that  is  developed  for  one 
analysis  cannot  be  used  easily  in  another  analysis  that 
requires  the  Scime  value  scales  but  contains  different  deci- 
sion options.  This  is  because  the  scales  are  defined  by  the 
particular  decision  options.  Since  in  this  case,  the  analysis 
was  expected  to  be  used  on  a recurring  basis  for  similar 
decision  problems  (S1123),  the  swing-weight  method  should 
« not  have  been  used. 

Alternatively,  more  effort  should  have  been 
spent  to  establish  an  absolute  scale  for  each  value  dimension 
that  could  be  used  for  subsequent  recurring  analyses.  For 
instance,  the  range  of  the  allied  goodwill  scale  may  have 
^ been  defined  as  the  difference  between  the  goodwill  at  the 
''  height  of  the  Vietnam  War  and  the  goodwill  right  after  World 
War  II. 


Another  drawback  to  the  swing -weight  method  is 
that  it  assumes  that  a linear  combination  of  the  different 
valuation  criteria  is  the  appropriate  way  to  combine  them 
into  a single  figure.  In  this  case,  this  linearity  condition 
did  not  hold  strictly,  but  the  cost  of  building  a non-linear 
model  was  large  (P222) , and  its  contribution  to  the  improved 

logic  of  the  analysis  was  small  (P114) . 

Note  that  this  analytic  choice  does  not  fit 
conveniently  into  a single  taxonomic  category  but  is  com- 
posed of  several  categories.  This  suggests  that  special 
taxonomies  may  be  appropriate  for  analytic  choices  as  well 
as  for  situations.  (These  special  situation  taxonomies  are 
discussed  in  Section  4,2.2  of  Volume  I.)  Such  special 
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taxonomies  would  contain  classifications  of  analytic  options 
such  as  "swing  weights,"  which  would  completely  describe 
analytic  methods  in  a form  that  is  readily  recognizable  to 
certain  observers,  but  would  require  different  taxonomies 
for  different  observers. 

The  ninth  analytic  choice  was  to  evaluate  the 
decision  options  directly  using  certainty  equivalents  {A311f)) 
rather  than  to  assess  probability  distributions  over  the 
outcomes  that  might  result  from  the  choice  of  those  options 
(for  all  value  dimensions  except  "oil  from  the  Mideastern 
country").  This  feature  required  the  value  assessor  to 
consider  implicitly  the  uncertainties  in  the  problem.  Thus, 
the  method  is  most  applicable  when  a single  person  has 
access  to  both  the  information  necessary  to  make  value 
judgments  and  information  pertinent  to  the  uncertainties  of 
the  situation.  Here,  this  condition  held  for  all  value 
judgments  except  those  related  to  the  dimension  "oil  from 
the  Mideastern  country." 

The  tenth  analytic  choice  v»as  to  elicit  the 
utility  function  for  "oil  from  the  Mideastern  country"  using 
direct  rating  (A4532)  rather  than  using  an  indirect  technique 
such  as  reference  gambles.  In  general,  the  reference-gamble 
technique  is  very  costly  (P222)  and  time  consuming  (P212)  to 
use  unless  the  decision  maker  is  very  familiar  with  decision 
analysis  (S24213) , which  was  not  the  case  here.  In  ^addition, 
unless  the  decision  maker  is  very  familiar  with  decision 
analysis,  he  is  likely  to  give  illogical  responses  (P114)  to 
the  questions  required  during  a reference-gamble  elicitation. 
Thus  the  use  of  the  reference-gamble  technique  was  not 
indicated  as  a primary  elicitation  technique  in  this  situation. 
However,  the  reference-gamble  framework  was  used  to  feed 
back  the  implications  of  the  direct  elicitation  to  the 
decision  maker  and  thus  to  refine  the  original  direct  elicita- 
tion. (This  refinement  used  the  reference  gamble  as  a null 
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inetrument.  That  is,  the  decision  maker  had  to  respond  only 
that  he  agreed  or  disagreed  with  the  implication  of  his 
direct  rating;  he  aid  not  have  to  respond  to  elicitation 
questions  with  a probability  or  a value.)  We  recoimend  this 
use  of  the  reference-gamble  technique  when  the  decision 
maker  has  a limited  familiarity  with  decision  analysis. 


2.4  Conclusions 


In  this  case,  the  taxonomic  framework  seems  to  serve  well 
the  purpose  of  determining  how  much  decision  analysis  to  use. 
However,  two  characteristics,  the  large  stakes  involved 
and  the  difficulty  of  the  choice,  would  have  been  sufficient 
to  indicate  that  a large  analysis  was  justified.  In  addition, 
a single  characteristic,  the  si;;e  of  the  analysis  budget, 
was  sufficient  to  determine  the  amount  of  analysis  that 
could  actually  be  done. 

For  other  analytic  choices,  however,  the  taxonomies,  at 
their  current  level  of  development,  do  not  appear  to  be  very 
helpful  in  general.  Even  though  most  of  the  reasoning 
surrounding  the  choices  can  be  explained  in  terms  of  elements 
in  the  taxonomies,  it  would  have  been  very  difficult  for 
most  analytic  choices  to  apply  the  taxonomies  directly  in 
the  course  of  the  unfolding  situation.  Such  an  application 
may  require  the  development  of  a sequential  matching  pro- 
cedure such  as  that  suggested  in  Section  4.2.1  of  Volur.ie  I. 

In  addition,  some  of  the  analytic  choices,  such  as  the 
choice  to  build  a partial  model  of  the  decision  options 
(A2223) , cannot  be  explained  easily  using  the  taxonomies  in 
their  present  form.  This  problem  may  be  overcome  by  devel- 
oping the  taxonomies  to  a much  finer  level  of  detail. 

However,  such  a development  is  outside  of  the  scope  of  the 
current  effort  and  may  even  prove  to  be  a completely  unmaiage- 
able  task.  Alternatively,  the  problem  may  be  overcome  by 
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the  development  o£  specific  taxonomies  (as  suggested  by 
Section  4.2.2  of  Volume  I).  Again,  the  development  of  such 
taxonomies  is  outside  of  the  scope  of  the  present  effort. 

On  the  other  hand,  even  the  current  development  of  the 
taxonomies  may  have  been  helpful  in  making  some  of  the 
specific  analytic  choices.  Specifically,  the  balance  of 
effort  may  have  been  improved  and  the  choice  to  use  certainty 
equivalents  may  have  been  facilitated  if  the  taxonomy  had 
been  available  to  the  decision  analysts. 

2. 5 Appendix;  Situation  Characterization 

This  appendix  characterizes  the  foreign  policy  case 
along  the  dimensions  of  the  situation  taxonomy  that  is 
described  in  Section  A of  Volume  III.  A summary  of  this 
characterization  is  presented  in  Table  2-2, 

1 Decision  Substance 


11  Basic  Situation 

1111  Current  Choice.  The  problem  was  to  decide 
immediately  on  a policy  recommendation. 

1123  Few  recurrences.  The  policy  advisor  expected 
to  make  recommendations  similar  to  this  one 
several  times  within  the  next  few  years. 

1131  Operating  act.  The  action  was  what  policy  to 
recommend,  not  whether  to  seek  information 
about  the  situation. 

12  Options 
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DECISION  SUBSTANCE  2 DECISION  PROCESS 


11 

BASIC  SITUATION 

21 

REACTION  TIME 

1111 

current  choice 

214 

months 

1123 

few  expected  recurrences 

1131 

operating  ect 

22 

ANALYTIC  PROCESSES 

12 

OPTIONS 

2215 

many  import  sources 

222121 

group  of  decision 

1211 

broad 

22223/ 

analysis  consultants 

1222 

fuzzy 

2230 

no  constraints  on  analytic  methods 

1236 

many  possible  choices 

2242 

specific  documentation  format 

1241 

radical 

2252 

training  project 

1262 

dynam  ic 

23 

ORGANIZATIONAL  PROCESSES 

13 

EASE  OF  DECISION 

23111 

spontaneous  initiation 

1313 

difficult 

23123 

decision  Initiated  by  superior 

1322 

unfamiliar  to  decision  maker 

23213 

recommendation 

1331 

option  generation  is  key 

2331 

moderate  coordination 

consideration 

2343 

pre-  and  post -decision  justification 

2352 

very  controversial 

14 

STAKES 

2361 

loose  control  of  decision  maker 

2372 

"rational  actor"  model 

1444 

maximum  option  impact 

23B1 

risk  averse 

greater  than  $10  million 

24 

DECISION  MAKER  CHARACTERISTICS 

15 

OUTCOME  VALUATION 

24114 

staff  member 

1512 

very  difficult  net  valuation 

24212 

some  familiar  with  decision  analysis 

1523 

many  value  dimensions 

1531 

approximate  metric 

25 

RESOURCES  AVAILABLE 

1641 

low  combinability 

1552 

medium  time  horizon 

2512 

ready  computer  availability 

1563 

difficult  component  valuation 

2522 

strong  staff 

2532 

decision  analysis  specialist 

16 

OUTCOME  UNCERTAINTY 

2542 

high  availability  of  decision  maker 

2552 

high  availability  of  assessors 

1613 

many  uncertainties 

2562 

modest  analysis  budget 

1621 

difficult  to  assess 

1632 

medium  amount  of  uncertainty 

1641 

low  impact  of  subsequent  acts 

1652 

lends  itself  to  indirect  techniques 

1662 

medium  hindsight 

Table  2-2 

SUMMARY  CHARACTERIZATION  OP  THE  FOREIGN  POLICY  DECISION  SITUATION 
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1211  Broad  options.  In  this  caso,  the  conunittee 
chairman  could  consider  anything  that  could 
be  done  from  a foreign  policy  standpoint  to 
improve  the  energy  situation  in  the  U.S. 

1222  Fuzzy  options.  There  were  many  ways  in  which 
the  options  could  be  specified.  For  instance, 
the  options  could  have  been  specified  in 
terms  of  complete  agreements,  that  is,  they 
could  have  specified  exactly  what  the  U.S. 
would  give  up  and  what  it  would  receive  in 
return.  On  the  other  hand,  the  options  could 
specify  the  U.S.  concession  and  treat  the 
return  as  an  uncertain  quantity. 

1236  Many  possible  options.  Since  the  task  was  to 
be  responsive  to  the  question,  "What  can  the 
US  do  with  foreign  policy  to  get  more  oil?" 
many  possible  options  existed.  Options  could 
have  ranged  between  agreements  and  unilateral 
concession,  two-party  and  multi-party  options, 
and  unidimensional  and  multi-dimensional 
options. 

1241  Radical.  Major  departures  from  the  existing 

U.S.  foreign  policy  would  need  to  be  considered. 

1252  Oyneunic.  A decision,  once  it  was  made,  could 
be  modified  by  later  decisions. 

13  Ease  of  Decision 

1313  Difficult.  The  correct  decision  was  not 
obvious  to  the  decision  maker  before  his 
problem  was  analyzed.  (However,  he  did  have 
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a rough  idea  of  the  limits  of  a reasonable 
solution. ) 

1322  Unfamiliar  decision.  The  decision  maker  was 
a broad-based  policy  advisor;  he  had  no 
previous  experience  in  making  foreign  policy 
recommendations . 

1331  Option  generation  was  the  key  consideration. 
Since  the  alternatives  were  so  broad  and 
fuzzy,  the  key  determinant  of  a good  decision 
was  the  generation  of  good  decision  options. 

Stakes 

1444  It  was  clear  from  the  beginning  that  the 
range  of  important  options  would  have  a 
difference  in  impact  on  the  order  of  several 
hundred  million  dollars  in  the  federal  budget. 

Outcome  Valuation 

1512  Valuation  was  very  difficult,  as  many  con- 
flicting intangible  valuation  criteria  needed 
to  be  considered. 

1523  Many  valuation  dimensions  were  important. 
Potentially,  everything  of  interest  to  the 
well-being  of  the  United  States  in  the  near 
term  and  in  the  long  term  was  important  in 
this  decision.  These  value  dimensions  were 
included,  but  were  not  necessarily  limited 
to:  volume  and  price  of  oil  from  the  Mid- 
eastern country,  balance  of  payments,  allied 
goodv;ill,  sentiments  within  the  U.S.,  and  oil 
from  other  countries. 


1531  Approximate  metric.  Some  of  the  valuation 
dimensions  posessed  natural  metrics,  such  as 
oil  price  (dollars) , and  some  did  not,  such 
as  allied  goodwill. 

1541  Low  combinability.  The  five  value  dimensions 
mentioned  in  1523  above  do  not  combine 
naturally. 

1552  Medium  timing  horizon.  Some  of  the  effects 
of  this  decision  could  extend  over  about  10 
years . 

1563  Difficult  component  valuation.  It  was  difficult 
to  evaluate  the  possible  outcomes  on  any  of 
the  valuation  dimensions  listed  in  1523 
above . 

Outcome  Uncertainty 

1613  Many  uncertainties  affected  the  outcome.  The 
impacts  of  the  decision  on  the  volume  and 
price  of  oil,  the  balance  of  payments,  allied 
goodwill,  and  sentiments  in  the  U.S.  were  all 
uncertain.  Furthermore,  each  of  these 
uncertain  impacts  was  comprised  of  several 
subsidiary  uncertainties. 

1621  All  of  the  uncertainties  were  difficult  to 
assess . 

1632  Medium  amount  of  uncertainty.  For  each  of 

the  uncertain  quantities,  its  relative  credible 

interval,  Median ' between 

.10  and  1.00. 
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1641  Subsequent  acts  were  relatively  unimportant 
to  this  decision. 

1653  The  uncertainties  lent  themselves  naturally 
to  indirect  judgment  techniques. 

1662  Medium  hindsight  monit'“ring.  Some  of  the 

outcomes  of  the  decision  could  be  determined 
promptly  and  accurately  and  others  could  not. 

Decision  Process 


21  Reaction  Time 

214  A recommendation  was  required  within  60  days 
of  when  the  analysis  was  first  considered. 

22  Analytic  Process 

2215  Several  sources,  including  reports,  experts, 
and  the  decision  maker,  would  provide  inputs 
to  the  analysis. 

22212  and  22223  A group  of  decision  analysis 

consultant  was  readily  available  to  perform 
the  analysis. 

2230  No  restrictions  constrained  the  analysis 
method. 

2242  The  final  report  of  the  policy  recommendation 
would  have  to  conform  to  a specific  format. 

2252  The  committee  chairman  was  very  interested  in 
trying  new  analytic  methods. 


23  Organizational  Processes 


23111  This  decision  was  initiated  spontaneously 

in  response  to  unanticipated  developments;  it 
was  not  scheduled  ahead  of  time. 

23123  The  committee  chairman's  superior,  the  high- 
level  policy  maker,  initiated  the  decision. 

23213  The  chairman  was  responsible  for  making  a 
recommendation  to  his  boss. 

2331  A moderate  degree  of  coordination  was  required 
between  this  decision  and  other  foreign 
policy  decisions. 

2343  The  reasons  for  the  foreign  policy  recommen- 
dation would  have  to  be  justified  both  before 
and  after  the  recommendation  was  made. 

2352  This  decision  was  a very  controversial  one. 

2361  The  chairman's  actions  were  only  loosely 
controlled. 

2372  The  "rational  actor"  model,  which  hypothesizes 
a single  decision  maker  who  relies  on  reasoned 
judgments  in  making  his  decisions,  was  met 
fairly  well  in  this  case.  The  policy-maker 
was  expected  to  base  his  action  on  a care- 
fully reasoned  and  logically  consistent  set 
of  arguments  treating  the  United  States  as  a 
single  entity. 
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2381  The  United  States  could  be  considered  to  be 
slightly  risk  averse  on  all  of  the  value 
dimensions. 

Decision  Maker  Characteristics 

24114  The  decision  maker,  the  committee  chair- 
man, was  a staff  member,  not  a line  manager. 

24212  The  chairman  had  some  familiarity  with 
decision  analysis. 

Resources  Available 

2512  A large  computer  facility  was  readily  avail- 
able. 

2522  A strong  staff  was  available  to  work  on  the 
analysis. 

2532  Decision  analysis  specialists  were  readily 
available. 


2542  The  decision  maker  was  available  on  each  of 

two  half-days  in  each  week  during  the  analysis, 

2552  Probability  and  value  assessors  were  readily 
available  throughout  the  analysis. 


2562  Modest  analysis  budget.  A budget  of  about 
$20,000  was  available  for  the  analysis. 


3.0  CASE  2:  A BUSINESS  INVESTMENT  DECISION 


3.1  Introduction 


This  case  refers  to  a decision  analysis  performed  over 
a two-week  period  in  February  1975  to  analyze  the  decision 
of  whether  or  not  a six-month  option  on  certain  patent 
rights  should  be  purchased.  A technical  description  of  this 
analysis  appears  in  Ulvila,  Brown«  and  Packard  (1977) > and 
it  is  not  reproduced  here. 

3.1.1  Problem  background  - Late  in  1974,  the  president 
of  the  AIL  Division  of  Cutler-Hammer,  Incorporated^  discussed 
with  an  independent  inventor  an  opportunity  to  invest  in  a 
new,  patented  flight-safety  product.  The  potential  product 
was  related  to  one  of  the  company's  major  areas  of  business 
although  not  to  a current  product  line,  in  a narrowly  defined 
sense.  Applications  could  be  to  both  civilian  and  military 
aircraft,  but,  in  keeping  with  AIL's  major  business  activity,  1 

only  the  defense  market  rights  were  considered.  Because  the  | 

product  would  be  technically  compatible  with  other  AIL  pro-  J 

ducts,  there  were  no  concerns  about  the  company's  engineering  j 

or  production  capabilities. 

The  opportunity  consisted  of  patent  rights  to 
this  flight-safety  system,  limited  to  the  defense  market.  A 
strong  patent  position  together  with  superior  technical 
performance  was  claimed  by  the  inventor.  Design  of  the  ! 1 

product  was  still  in  the  development  stage,  but  it  was 
developed  enough  to  permit  an  evaluation  of  all  probable 


Cutler-Haunmer  is  an  electronics  corporation  with  annual 
sales  of  about  $400  million.  The  AIL  division's  annual 
sales  are  about  $100  million. 


costs.  The  inventor  would  make  information  on  the  design 
available,  but  AIL  would  have  to  undertake  packaging  for 
military  aircraft.  The  immediate  decision  to  be  made  was 
whether  or  not  to  pay  $50,000  to  purchase  an  option  on  these 
rights  for  a period  of  about  six  months,  at  which  time  a 
second  decision  would  have  to  be  made  on  a substantial  pre- 
payment against  future  royalties  in  order  to  support  con- 
tinued development.  After  initially  satisfying  himself  about 
the  possible  business  potential,  the  president  of  AIL  assigned 
a small  task  team  the  responsibility  of  fully  investigating 
the  opportunity  and  developing  a recommended  plan  of  action. 

One  of  the  potential  difficulties  in  making  this 
decision  concerned  jurisdictional  areas  affecting  flight- 
safety  standards.  These  may  be  set  variously  by  the  Military 
Services  and  the  Federal  Aviation  Administration  (FAA) . 
Although  military  aircraft  must  comply  with  many  FAA  rulings 
during  peacetime,  they  need  not  comply  with  all  of  them,  and 
they  may  not  comply  with  most  of  them  during  wartime.  In 
departing  from  FAA  rulings,  the  Department  of  Defense  (DoD) 
would  be  more  likely  to  set  more  stringent  standards  during 
peacetime  and  less  stringent  ones  during  wartime. 

In  addition,  the  long  history  of  flight  safety  is 
one  of  inaction  by  the  FAA  until  a major  accident  forces  a 
corrective  response.  Nevertheless,  an  FAA  ruling  on  a 
relevant  requirement  was  scheduled  for  the  near  future,  but 
it  was  doubted  that  the  FAA  would  make  its  decision  according 
to  schedule."'  einc^  the  inventor  wished  to  make  the  offer  to 
other  companies  if  AIL  were  not  interested,  he  requested  that 
the  decisioir  on  the  purchase  of  the  option  be  made  in  a 
matter  of  a few  weeks,  a period  of  time  clearly  inadequate  to 
resolve  any  of  the  identified  uncertainties. 
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Not  the  least  of  these  was  that  there  were  several 
alternative  technical  solutions  to  the  flight-safety  problem. 
Depending  on  the  specific  details  of  the  FAA's  decision^  one 
or  another  of  these  competing  solutions  would  be  most  favored. 
Because  the  inventor's  particular  design  was  controversial, 
there  was  no  assurance  that  it  would  receive  favorable 
treatment  even  with  a secure  patent  position. 

3.1.2  Utilization  of  decision  analysis  - Formal  deci- 
sion analysis  had  not  been  previously  used  at  AIL  to  make 
business  decisions.  However,  AIL's  director  of  long-range 
planning  was  familiar  with  the  theory  and  its  typical  appli- 
cations through  the  literature.  In  addition,  AIL  was  inter- 
ested in  trying  these  techniques  on  an  actual  decision  problem 
in  order  to  evaluate  their  worth.  The  decision  on  patent 
rights  was  considered  to  be  an  appropriate  trial  application 
of  decision  analysis. 

The  task  team  pursuing  this  objective  consisted 
of  AIL  personnel  who  reported  to  the  vice  president  for 
business  development  and  analysts  from  Decisions  and  Designs, 
Incorporated  (DDI) . AIL  staff  provided  the  substantive 
knowledge  required  for  the  analysis  and  the  DDI  analysts 
provided  the  methodology.  Throughout  the  two-week  period  of 
analytic  effort,  there  was  continual  interaction  between  the 
key  decision  makers  in  AIL's  top  management  and  the  analysts. 

An  initial  modeling  effort  structured  and  quan- 
tified the  main  elements  of  value  and  uncertainty.  This 
initial  quantification  slightly  favored  purchasing  the  option 
because  of  its  expected  value  of  discounted  earnings.  A 
further  analysis,  however  indicated  that  this  model  was  very 
sensitive  to  some  of  the  values  used  initially.  It  beccune 
obviously  important  to  analyze  these  values  in  far  greater 
detail  to  develop  confidence  in  them.  Sub-modeling  parts  of 


the  analysis  provided  a way  to  gain  this  confidence  and  under- 
standing. During  the  refinement  process,  a new,  secondary 
valuation  criterion,  return  on  capital,  was  identified;  but  it 
proved  to  have  little  effect  on  the  result.  When  this  refined 
analysis  was  completed,  it  no  longer  supported  purchasing  the 
option. 


The  presentation  of  this  analysis  to  the  division 
president  increased  his  understanding  of  the  problem,  and  the 
discussion  that  arose  during  the  presentation  revealed  a new 
alternative,  waiting  and  seeking  a sub-license.  A final  ex- 
amination and  refinement  of  the  model  indicated  a slightly 
higher  expected  value  for  purchasing  the  option  over  waiting. 

This  slight  difference  in  expected  value  led  to  a 
consideration  of  other  factors.  Specifically,  attitude  toward 
risk  was  introduced  into  the  analysis  by  excimining  not  only  the 
expected  value  of  the  option,  but  also  the  distribution  of 
possible  values.  Since  the  slight  difference  favoring  the 
purchase  of  the  option  resulted  from  a high  return  with  a low 
probability  and  was  accompanied  by  a high  probability  of  sig- 
nificant loss,  the  division  president  realized  that  the  high- 
risk  nature  of  the  opportunity  required  special  consideration. 
In  this  case,  it  ultimately  determined  the  decision  not  to 
purchase  the  option.  The  alternative  action  of  waiting  re- 
sulted in  lower  expected  and  potential  gain,  but  it  also 
eliminated  the  potential  loss.  Thus,  it  became  the  preferred 
alternative. 


In  the  final  stages  of  the  decision-making  process, 
the  analysis  served  as  a vehicle  to  focus  discussion  on  the 
critical  issues  surrounding  the  decision  and  to  minimize  the 
consideration  of  irrelevant  issues.  When  the  results  of  the 
analysis  were  presented  to  the  president  of  AIL  and  his  vice 
presidents  for  business  development  and  operation,  they 
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eventually  agreed  to  reject  the  option,  wait,  and  possibly  seek 
a sub-license  at  a later  time.  This  decision  to  wait  was 
reached  easily  and  unanimously,  without  reservation  on  the  part 
of  any  of  the  participants. 

3. 2 Performance  Determinants  of  a Good  Analysis 

In  this  particular  case,  the  situation  dictated  that  a 
good  analysis  was  one  that  performed  well  in  four  performance 
categories.  The  analysis  had  to  be  fast,  logical,  inexpensive, 
and  easy  to  explain. 

2 

The  short  time  available  for  the  analysis  (S213)  dictated 
that  the  overall  elapsed  time  of  the  analysis  (P2141)  was  the 
most  important  measure  of  a good  analysis.  Three  components  of 
the  overall  analysis  time  were  important.  In  decreasing  order 
of  importance,  these  components  were  that  the  time  spent  struc- 

3 

turing  the  decision  model  had  to  be  kept  low  (P2111) , the 
values  and  probabilities  ^ad  to  be  assigned  quickly  (P2121) , 
and  the  calculations  in  the  analysis  had  to  be  performed 
quickly  (P2131) . 

Because  the  choice  of  the  best  option  was  the  most  impor- 
tant consideration  in  this  case  (S1334)  and  the  choice  was  a 
moderately  difficult  one  {S1312),  the  quality  of  the  reasoning 
and  logic  contained  in  the  analysis  (P114)  was  the  second  most 
important  measure  of  a good  analysis.  Four  components  of  the 
quality  of  logic,  listed  below  in  decreasing  order  of  sig- 
nificance, were  important.  First,  the  analysis  had  to  be 


2 

A complete  characterization  of  this  decision  situation  is 
presented  in  Section  3.5  below. 

3 

The  parenthetical  references  that  are  prefaced  with  "P"  are 
performance  measures,  which  are  examined  in  Section  C of 
Volume  III. 
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complete  in  its  scope  (P115) ; it  had  to  address  the  complete 
aecision  problem  rather  than  a part  of  the  problem,  such  as  the 
assessment  of  probability.  Second,  it  was  important  for  the 
analysis  to  be  conceptually  complete  (Pill) ; it  must  take  all 
important  considerations  into  account  and  avoid  serious  errors 
of  approximation.  Third,  it  was  important  for  the  analysis 
method  to  make  sound  inferences  from  the  available  data  (P113) . 
Fourth,  it  was  important  for  the  analysis  to  allow  the  problem 
to  be  divided  into  manageable  8ub~problems  (P112) . 

4 

Because  the  stakes  involved  in  the  decision  were  low 
(S1432,  S1442)  and  the  analysis  budget  was  low  (S2561) , the 
third  most  important  determinant  of  a good  analysis  was  its 
cost.  The  overall  cost  of  the  analysis  had  to  be  low  (P2241) . 


Finally,  because  the  responsibility  for  the  decision  was 
dispersed  among  a small  group  (S23221) , the  fourth  important 
performance  measure  was  the  quality  with  which  the  decision  and 
its  supporting  arguments  could  be  communicated  (P321) . 

3 . 3 Analytic  Choices 

The  recommended  analytic  options  for  this  situation  are 
summarized  on  Table  3-1.  The  following  sections  explain  the 
reasoning  behind  the  most  important  recommendations  and  provide 
an  indication  of  the  order  in  which  the  analytic  choices  might 
be  made.  Although  the  discussion  below  is  meant  to  be  self- 
explanatory,  the  reader  may  wish  to  refer  to  Section  3 of 
Volume  III  for  a more  detailed  description  of  the  range  of 
analytic  choices  that  were  available. 

I 

3.3.1  Choice  of  the  amount  of  analysis  - Guidelines  of  S 

situational  characteristics  that  favor  using  decision  analysis  j 

are  presented  in  Section  3.1.2  of  Volume  I.  This  case  can  be  1 

characterized  along  these  situational  dimensions  as  follows.  j 

i 

A 

(A 
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USER'S  OPTIONS 


4  INPUT  SPECIFICATION 


11  USE  DECISION  ANALYSIS  AT  ALL? 

112  use  decision  analysis 

12  DOLLAR  AMOUNT  OF  ANALYSIS 

1 21  snnall  amount  of  analysis 

13  ROLE  OF  DECISION  ANALYSIS 

1311  privoie  aid 
1322  optional  decision 
1332  current  decision 
1342  display 
1351  communication 

14  ORGANIZATION 

1419  consultants  performed  the  analysis 
1423  experts  provided  inputs 
1431  close  relationship  with  decision 
process 

15  RESOURCES 

1511  decision-analytic  expert 
1521  computer 

2 MODEL  APPROACH  OPTIONS 

21  COMPLEXITY  OF  ANALYSIS 

211  simple 

22  COMPREHENSIVE  /PARTIAL  ANALYSIS 

221  comprehensive 

23  DEGREE  OF  APPROXIMATION 

233  exact 

3 INPUT  STRUCTURE 

31  UNCERTAINTY 

31 12  complete  probability  distributions 
3122  medium  time  horizon 
3132  act  modeled  as  an  event 
3142  medium  level  of  detail 

3151  low  grouping  of  uncertain 

quantities 

32  VALUE 

3221  modest  decomposition 
32319  aggregation 


41  DECISION  OPTIONS 

4111  real  option 

42  EVENTS 

422  specific 

43  VALUE  CRITERIA 

4311  natural  units 

4322  floating-zero  base 

4331  present  value  and 

4333  time  flow 

44  INDIRECT  ASSESSMENTS 

4411  few  conditioning  tiers 
4431  decomposed  assessment  model 

45  ELICITATION  TECHNIQUE 

4612  I'i.ect  elicitation  of  discrete 
probabilities 

4521  fractile  elicitation  of 

continuous  probabilities 

5 OUTPUT 

51  SPECIFICATION 

5131  value  distributions 
62  DISPLAY  FORMAT 
B211  graphic  display 
53  ANALYTIC  DEVICES 
6391  grouping 

6 MODEL  MANAGEMENT 

81532  a moderate  amount  of 
sensitivity  analysis  on 
both  values  and  probabilities 


Table  3-1 

SUMMARY  OF  AIL  ANALYSIS  CHOICES 
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Important  factors  favoring  a large  amount  of 

analysis : 

o The  options  were  clearly  defined  (51221) . 
o Choice  was  the  key  consideration  (S1334) . 
o This  analysis  was  considered  a "trial  run"  of  deci- 
sion analysis,  and  it  might  determine  whether  deci- 
sion analysis  would  be  used  in  the  future  (S2252) . 
o The  decision  maker  was  somewhat  familiar  with 
decision  analysis  (S24211) . 

Important  factors  against  a large  amount  of  analy- 
sis: 

o The  choice  was  not  a very  difficult  one  (S1312) . 
o The  stakes  were  low  (S1421,  S1432,  S1442) . 

o Few  input  sources  needed  to  be  considered  (S2211  or 

S2212)  . 

o The  decision  maker  did  not  need  to  justify  his  deci- 
sion to  others  (S2340) . 

Other  factors  favoring  decision  analysis: 

o The  decision  was  a current  one  (Sllli)  . 
o The  "rational  actor"  model,  which  hypothesizes  a 

single  decision  maker  who  relies  on  reasoned  judg- 
ments in  making  his  decisions,  fits  very  well  in  this 
case  (S2372) . 

o A large  computer  facility  was  readily  available 
(S2512) . 

o A Strong  staff  was  available  (S2522) . 

Other  factors  against  decision  analysis; 
o The  choice  was  not  expected  to  recur  (S1122) . 
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o Outcomss  could  be  easily  evaluated  (S1510) . 
o A decision  was  required  in  a few  days  (S213) . 
o The  decision  maker  could  spend  only  a little 
time  on  the  analysis  (S2532) . 
o Only  a small  budget  was  available  for  the  analysis 
(S2561) . 

More  factors  argued  against  using  decision  analysis 
in  this  case  than  argued  for  it.  However,  the  fact  that  the 
proposed  decision  analysis  would  serve  as  a "trial  run"  to 
determine  whether  or  not  decision  analysis  would  be  used  for 
future  decisions  was  a very  strong  argument  in  favor  of  the 
analysis.  In  addition,  certain  of  the  favorable  situation 
characteristics  tended  to  negate  some  of  the  unfavorable  ones. 

For  example,  because  the  options  were  clear,  the  decision  maker 
was  somewhat  familiar  with  decision  analysis:  and  because  a strong 
staff  was  available,  an  analysis  could  be  made  relatively 
quickly  and  inexpensively.  Thus,  the  importance  of  a short 
reaction  time  and  a small  analysis  budget  was  reduced.  Over- 
all, then,  some  analysis  was  justified  (A112) . 

The  two  situation  characteristics  that  were  the 
most  important  determinants  of  the  amount  of  analysis  were  the 
stakes  and  the  size  of  the  analysis  budget.  In  this  case,  both 
the  stakes  and  analysis  budget  were  small,  and  thus  only  a 
small  decision  analysis  was  indicated  (Al2,l)  . 

3.3.2  Type  of  analysis  - The  choice  of  whether  to  do 
decision  emalysis  at  all  and  the  general  level  that  such  an 
analysis  should  take  are  decisions  that  can  be  made,  for  the 
most  part,  by  the  decision  maker  without  involving  the  decision 
analyst.  The  choice  of  the  type  of  analysis  to  use,  however, 
requires  the  judgment  of  the  analyst  as  well  as  that  of  the 
decision  maker.  In  this  case,  analytic  choices  were  made  at 
roughly  four  points  in  the  analysis:  before  formal  modeling 
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commenced,  during  the  initial  modeling  effort,  during  the  final 
modeling  effort,  and  immediately  preceding  the  presentation. 
(Please  note  that  the  discussion  below  is  more  complete, 
covering  more  analytic  choices,  than  similar  discussions  in  the 
other  case  studies.) 

Before  formal  modeling  - At  the  first  point  of 
analytic  choice,  before  formal  modeling  commenced,  the  decision 
maker  and  the  analyst  both  affected  the  choice  of  analytic 
options.  Five  major  analytic  choices  were  made  at  this  time. 

First,  the  decision  maker  chose  to  use  decision  analysis  con- 

\ 

sultants  in  a close  working  relationship  with  AIL  staff  mem- 
bers. Second,  the  decision  maker  and  analyst  decided  to  model 
the  problem  from  the  personal  point  of  view  of  the  decision 
maker  and  to  emphasize  the  display  of  the  analysis.  Third,  the 
analysts  decided  to  build  a simple  but  comprehensive  model  of 
the  decision  problem  and  to  refine  the  model's  structure  on  the 
basis  of  a sensitivity  analysis.  Fourth,  the  decision  maker 
and  analyst  decided  to  use  the  judgments  of  AIL  staff  members 
as  inputs  to  the  model.  Fifth,  the  analyst  decided  to  use  a 
computer  to  perform  the  calculations  in  the  model. 

Decision  analysis  consultants  were  indicated  for 
this  analysis  {A1419,  A1511)  because  AIL  had  no  previous 
experience  using  decision  analysis,  it  did  not  have  a well- 
developed  in-house  decision  analysis  capability  (S24212) , and 
decision  analysis  consultants  were  readily  available  (S2532) . 
The  close  working  relationship  between  the  decision  makers  and 
the  AIL  staff  members  (A1431)  is  a relationship  that  we  recom- 
mend in  general.  We  only  recommend  against  this  relationship 
when  the  decision  is  a small,  repetitive  one  which  does  not 
require  much  judgment  on  the  part  of  the  decision  maker. 

The  decision  to  model  the  problem  from  the  decision 
maker's  (division  president's)  point  of  view  rather  than  from 


an  institution's  (Cutler-Hammer's)  point  of  view  (A1312)  was 
made  so  that  the  analysis  would  conform  to  the  decision  maker's 
motivation,  in  addition,  a model  that  is  built  to  resemble  the 
decision  maker's  personal  perception  of  the  problem  closely  is 
more  likely  to  yield  a decision  that  will  guide  his  action  than 
one  that  is  not.  The  decision  to  emphasize  the  display  aspect 
of  the  analysis  (A1342) , rather  than  the  optimization  aspect, 
is  one  that  we  recommend  in  general.  This  choice  was  par- 
ticularly important  in  this  case  because  several  people  shared 
the  responsibility  for  various  aspects  of  the  analysis  (S23221) . 
In  such  a case,  the  communication  of  the  reasons  surrounding 
the  choice  of  a decision  option  is  very  important  (P321) , and  an 

emphasis  on  the  display  of  the  analysis  promotes  such  communi- 
cation. 


Because  the  speed  (P2151) , logic  (P114) , cost 
(P2241) , and  communication  {P321)  of  the  analysis  were  so 
important,  a simple  but  comprehensive  model  was  indicated.  A 
simple  (A211)  model  can  usually  be  built  quickly  and  inexpen- 
sively, and  it  facilitates  communication.  However,  such  a 
model  needed  to  be  comprehensive  (A221) , encompassing  nearly 
all  relevant  considerations,  in  order  to  yield  the  completeness 
(Pill)  necessary  for  a logical  choice.  In  this  particular 
case,  a simple  yet  coraprehensive  analysis  was  possible  because 
the  options  under  consideration  were  clearly  and  narrowly 
defined  (S1221,  S1212) , the  value  dimensions  were  few  and  easy 
to  assess  (S1522,  S1561) , and  the  uncertainties  were  moderately 
easy  to  assess  (S1622) . Refining  the  analysis  based  on  the 
results  of  a sensitivity  analysis  (A6153)  was  the  fastest  and 
most  economical  way  to  ensure  that  the  final  model  would  be  the 
best  one  that  was  possible  within  the  time  available  for  the 
analysis. 


AIL  staff  members  were  the  only  sources  used  to 
provide  judgments  on  the  inputs  (A1423)  of  values  and  proba- 
bilities because  they  were  the  best  sources  available  within 
the  time  limit. 
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Finally/  a computer  program  was  written  specifi- 
cally for  this  problem  (A15212)  because  this  was  the  only  way 
that  the  sensitivity  of  the  decision  to  the  model's  inputs 
could  be  examined  quickly.  Furthermore,  because  the  initial 
model  did  not  contain  a large  decision  tree  structure  and  did 
not  use  complicated  probability  distributions,  but  did  require 
an  expected  present  value  calculation,  it  was  easier  to  write  a 
FORTRAN  program  for  this  model  than  to  use  any  of  the  available 
pre-packaged  decision  analysis  programs. 

During  initial  modeling  - During  initial  modeling 
most  of  the  analytic  choices  were  made  by  the  analyst.  The 
most  important  analytic  choices  made  at  this  time,  as  explained 
below,  concerned  the  time  horizon,  the  valuation  criteria,  and 
the  model  of  the  event  structure. 

A 10-year  time  horizon  (A3122)  was  indicated  for 
the  model  because  the  effects  of  the  immediate  decision  were 
expected  to  extend  over  this  length  of  time  (S1552) . Further- 
more, these  effects  were  easy  to  identify,  assess,  and  model. 

The  dec  i s i on^malter  ' iffd  i c a tea”  tHab  the  Warnings 
flows  and  return  on  capital  (ROC)  flows  over  the  10-year  time 
horizon  were  the  most  important  criteria  in  making  his  deci- 
sion. These  criteria  were  most  easily  understood  and  assessed 
in  their  natural  units  (A4311} , dollars  for  earnings  and  per- 
cent for  ROC.  In  addition  to  these  natural  units,  a "floating- 
zero"  base  (A4322),  which  evaluates  ail  outcomes  relative  to 
those  resulting  from  the  choice  of  a particular  alternative  (in 
this  case,  the  alternative  "do  not  purchase  the  option") , was 
also  appropriate  in  this  case.  The  floating  zero  base  was 
indicated  mainly  because  the  alternative  of  not  purchasing  the 
option  ("business  as  usual")  was  a convenient  benchmark  for 
comparison.  In  addition,  this  project  was  not  big  enough 
(S1432)  to  substantially  Influence  a large  part  of  a relevant 
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absolute  or  "fixed-zero"  scale  such  as  the  AIL  division's  total 
earnings. 


Flows  over  time  are  often  difficult  for  the  deci- 
sion maker  to  compare  directly,  thus  we  generally  recommend 
the  use  of  summary  measures  instead  of  flows  over  time  whenever 
such  summary  measures  are  available.  In  this  case,  the  earnings 
flows  over  time  combine  naturally  into  a present  value  by  using 
a discount  factor  tS1543) . In  addition,  it  was  AIL's  conven- 
tion to  evaluate  projects  such  as  this  on  the  basis  of  the 
present  value  of  their  pre-tax  earnings,  discounted  at  20%  per 
annum  (S2239) . These  situations  indicated  that  the  earnings 
flows  should  be  aggregated  into  a single  present  value  figure 
(A32319,  A4331) . On  the  contrary,  such  a convenient  aggre- 
gation method  does  not  exist  for  the  ROC  flows.  Furthermore, 

ROC  flows  do  not  combine  naturally  with  earnings  flows  (for  one 
thing  dollars  do  not  combine  naturally  with  percentages) . 

Thus,  aggregations  of  the  ROC  flows  into  a single  figure  and 
ROC  flows  with  earnings  flows  were  not  indicated. 

While  we  eventually  determined  that  several  uncer- 
”taiht:ieS“weTe  important,  we  initially  sought  to  maintain  the 
analysis  at  a very  simple  level  (A211) . Thus,  only  very  few 
events  should  have  been  modeled  in  the  early  analysis.  The 
subsequent  act  of  whether  or  not  to  exercise  the  option  and 
enter  into  a license  agreement  with  the  jinventor  was  Eunong  the 
items  that  were  included  in  the  early  analysis-.  However,  we 
recommended  that  this  subsequent  act  be  treated,  in  the  model, 
as  an  uncertain  event,  rather  than  as  an  act  (A3132) . There 
were  two  reasons  for  this  recommendation.  First,  this  subse- 
quent act  was  critical  to  the  outcome  of  the  initial  decision 
to  purchase  the  option  (S164) . Thus  the  subsequent  act  deserved 
some  explicit  modeling.  Second,  this  initial  decision  would 
result  in  a complex  information  search.  In  this  case  it  was 
convenient  to  avoid  modeling  explicitly  the  information  that 
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might  be  received  by  treating  the  subsequent  action  as  an 
uncertain  event. 

During  final  modeling  Analytic  choices  during  the 
final  modeling  were  made  by  the  analysts.  At  this  stage  in  the 
analysis,  the  most  important  analytic  choice  concerned  the 
direction  in  which  the  analysis  should  be  refined.  Two  main 
alternative  refinements  were  available.  The  event  and  earnings 
distributions  could  be  modeled  in  more  detail  or  the  ROC  valua- 
tion criterion  could  be  investigated  in  greater  detail.  The 
results  of  a sensitivity  analysis  (A61532)  indicated  that  a 
refinement  of  the  probability  distributions  of  events  and 
earnings  would  significantly  increase  the  logic  of  the  analysis 
for  only  a small  investment  of  time  and  effort.  On  the  other 
hand,  a cursory  investigation  indicated  that  a refinement  of 
the  ROC  valuation  criterion  would  only  slightly  increase  the 
logic  of  the  analysis  but  would  require  a large  investment  of 
time  and  effort.  Thus,  refinements  in  the  models  of  the  events 
and  earnings  were  indicated.  In  particular,  the  sensitivity 
analysis  indicated  that  the  decision  was  most  sensitive  to  the 
probability  of  winning  a contract  and  the  size  of  the  contract, 
and  thus  these  were  the  events  whose  models  shoul’  be  refined. 

The  probability  of  receiving  a contract  was  the 
most  difficult  probability  to  determine  because  it  depended  on 
four  things : whether  a plane  crash  would  occur  in  the  near 
future,  when  Congress  or  the  FAA  would  mandate  a flight-safety 
system,  whether  the  mandate  would  include  certain  standards, 
eind  whether  tough  competition  would  develop.  Since  the  depen- 
dence relationship  amcng  these  four  events  was  so  clear  (31653) 
and  each  event  was  so  easy  to  assess  (S1622) , the  analyst  could 
quickly,  accurately,  and  inexpensively  model  the  probability  of 
receiving  a contract  in  a conditioned  assessment  model  that 
contained  the  four  tiers  of  events  (A4411,  A3151) . 
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In  the  initial  model/  the  earnings  flows  were 
obtained  by  direct  judgment.  However,  the  assessors  found  it 
easier  to  think  about  earnings  if  they  decomposed  their  assess- 
ment into  three  components,  the  number  of  units,  the  price  per 
unit,  and  the  earnings  as  a percentage  of  sales  revenv\e  (S165,3)  . 
Thus,  a decomposed  assessment  model  (A3221)  was  the  beat  way  to 
refine  the  earnings  model.  In  this  model,  a continuous  dis- 
tribution was  assessed  for  each  of  the  three  components  of 
earnings  using  the  fractile  method  (A4521) . Although  this 
assessment  method  generally  provides  very  accurate  assessments, 
it  seldom  yields  distributions  that  have  analytical  forms,  such 
as  normal  distributions.  Thus,  the  assessed  distributions 
needed  to  be  modified  before  they  could  be  incorporated  into 
the  analysis.  Since  little  time  was  available  for  the  analysis 
(S213) , the  distributions  were  simplified  into  three-event  repre- 
sentations (A5391)  using  the  procedure  explained  on  page  189  of 
Brown,  Kahr,  and  Peterson  (1974).  In  addition,  it  appeared 
that  a special-purpose  computer  program,  TREE,  should  be  used 
to  perfoxmi  the  calculations  on  the  earnings  distributions 
(A15211) . At  the  time,  it  appeared  that  this  computer  program, 
which  interactively  elicits  a decision  tree  and  then  auto- 
matically processes  the  tree  and  presents  s distribution  of 
values,  could  provide  the  needed  calculations  more  quici ly  and 
accurately  than  the  analyst  could  do  manually.  However,  in 
retrospect  it  appears  that  an  analyst  could  have  done  a more 
efficient  job  of  calculation  than  the  computer  did?  thus, 
this  computer  program  should  not  have  been  used. 

Prior  to  Presentation  - Iimnediately  prior  to  the 
preser/,tation  of  the  analysis,  both  the  decision  maker  and  the 
analyst  helped  determine  the  proper  display  of  the  analysis. 

Tills  display  contained  two  important  elements;  a value  dis- 
tribution was  presented  for  each  decision  option  and  graphic 
displays  were  used  throughout  the  presentation. 
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A value  distribution  for  each  decision  option 
displayed  the  risk  inherent  in  the  decision.  Since  the  risk 
aversion  of  the  decision  maker  (S2381)  ultimately  became  the 
deciding  factor  of  the  analysis,  an  incorporation  of  risk  was 
very  important  to  the  analysis.  Because  the  decision  maker  had 
only  some  familiarity  with  decision  analysis  (S24211)  and 
because  the  amount  of  analysis  time  was  small  (S213)  and  the 
communication  of  the  analysis  was  important  (A1351) , a closed- 
form  utility  analysis  was  not  indicated.  Hoivever,  the  display 
of  a risk  profile  of  each  decision  option  (A5131)  was  indicated 
for  the  reasons  above  and  because  the  risk  profile  clearly 
highlighted  the  difference  between  the  two  options.  Specifi- 
cally, one  option  offered,  in  present-valued  earnings,  an 
expected  value  of  $100,000,  a large  chance  of  a loss  of  $50,000 
to  $200,000,  and  a small  chance  of  a gain  of  $5.25  million, 
while  the  other  option  offered  an  expected  value  of  $50,000, 
virtually  no  chance  of  a loss,  and  a small  chance  at  a gain  of 
$800,000. 

A graphic  display  of  the  results  of  the  analysis 
was  important  because  the  model  incorporated  time  flows  (A4333) 
and  a risk  profile  (A5131) , both  of  which  are  best  displayed 
graphically.  In  addition,  the  graphic  display  promoted  the 
communication  of  the  reasons  for  the  decision  (A1351)  by 
showing  how  the  various  events,  especially  in  the  conditioned 
assessment  (A4411) , interacted. 

3.4  Conclusions 

The  primary  motivating  objective  for  developing  the 
taxonomies  of  performance  measures,  analytic  options,  and 
situations  is  to  enable  a decision  maker  and  an  inexperienced 
analyst  to  make  the  types  of  analytic  choices  described  in 
Section  3 more  easily.  It  is  not  clear  that  the  present 
development  of  the  taxonomies  meet  this  objective  completely. 


The  taxonomic  framework  seems  to  serve  well  the  purpose  of 
aiding  in  the  broad  choices  of  whether  or  not  to  use  decision 
analysis  and  how  much  analysis  to  use.  For  these  choices,  the 
decision  maker  can  easily  use  the  taxonomies  by  characterizing 
his  decision  situation  along  the  primary  and  secondary  dimen- 
sions listed  in  Section  3.3.1  and  by  examining  the  degree  to  which 
his  situation  indicates  that  a large  amount  of  analysis  should 
be  used. 

However,  the  taxonomic  framework  does  not  appear  to  be  as 
helpful  for  making  more  detailed  analytic  choices,  such  as  the 
choice,  explained  in  Section  3.3.1  above,  of  how  to  refine  a 
simple  initial  analysis.  For  such  choices,  it  is  possible  for 
an  analyst  to  force  his  reasons  for  making  a choice  into  the 
framework  of  the  taxonomy,  but  it  is  not  straightforward  for 
him  to  do  so.  Because  such  reasoning  is  tricky  to  fit  into  the 
taxonomic  framework,  the  value  of  the  taxonomy  for  making  these 
choices  is  not  clearly  evident. 

3. 5 Appendix;  Situation  Characterization 

Section  3.3  above  explaines  the  major  analytic  choices 
that  needed  to  be  made  in  this  case.  This  section  presents  a 
more  complete  and  formal  characterization  of  the  situation  in 
terms  of  the  Situation  Taxonomy  that  is  specified  and  defined 
in  Section  A of  Volume  III.  Table  3-2  summarizes  this  charac- 
terization and  the  following  discussion  illustrates  each  ele- 
ment of  the  characterization  in  terms  of  specific  instances  in 
the  case. 

1 Decision  Substance 

11  Basic  Situation 
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1 DECISIOM  SUBSTANCE  2 DECISION  PROCESS 


11  BASIC  SITUATION  21  REACTION  TIME 


1111 

current  choice 

213 

days 

1122 

one  expected  occurrence 

1131 

operating  act 

22 

ANALYTIC  PROCESSES 

12 

OPTIONS 

2211/2 

few  input  sources 

22212) 

group  of  decision 

1212 

narrow 

22223/ 

analysis  'onsultants 

f 

1221 

clear 

2231 

few  restrictions  on  the 

|. 

1231 

two 

analytic  method 

: 

1241 

adaptive 

2240 

no  restrictions  on  the  reporting 

i 

12n 

static 

procedure 

2252 

"trial  run”  analysis 

13 

EASE  OF  DECISION 

23 

ORGANIZATIONAL  PROCESSES 

1312 

moderately  difficult 

• 

1321 

familiar  to  the  decision  maker 

23111 

spontaneous  initiation 

1334 

choice  is  key  consideration 

23121 

initiated  by  decision  maker 

23221 

a small  group  shared  responsibility 

5 14 

STAKES 

for  the  decision 

L- 

2330 

no  coordination 

1421 

low  cost  swing 

2340 

no  justification 

1432 

few  million  S value  swing 

2360 

uncontroversial 

1442 

few  hundred  thousand  S 

2361 

loose  performance  control  of 

option  impact 

the  decision  maker 

1 

2372 

rational  actor  model 

t 15 

P 

OUTCOME  VALUATION 

2381 

risk  averse 

*: 

1510 

easy  net  valuation 

24 

DECISION  MAKER 

1522 

low  no.  of  value  dimensions 

CHARACTERISTICS 

1532 

exact  metrics 

1 

1543 

high  natural  combinability 

24113 

high  level  authority 

i. 

1552 

medium  time  horizon 

24211 

little  familarity  with  decision 

1 

1561 

easy  component  valuation 

analysis 

1 16 

OUTCOME  UNCERTAINTY 

25 

RESOURCES  AVAILABLE 

1 

1613 

many  uncertainties 

2512 

computer  readily  available 

1 

1622 

moderately  assessable 

2522 

strong  staff 

1 

1632 

medium  amount  of  uncertainty 

2532 

decision  analysis  specialist 

|. 

1643 

important  subsequent  act 

2541 

littie  decision  maker  availability 

1' 

1653 

lends  itself  to  indirect  techniques 

26512 

high  availability  of  assessors 

1 

1661 

low  hindsight  monitoring 

25522 

high  sophistication  of  answers 

1 

2561 

small  analysis  budget 

1111  Current  choice.  The  problem  was  to  decide 
immediately  whether  or  not  to  purchase  the 
option. 

A 

1122  The  decision  was  expected  to  be  made  only  once. 

1131  Operating  act.  The  action  was  whether  or  not  to 
purchase  the  option,  not  whether  to  seek  infor- 
mation about  the  opportunity. 

12  Options. 

1212  Narrow  options.  The  options  were  defined  nar- 
rowly as  whether  or  not  to  buy  the  option  on  the 
patent  license. 

1221  Clear  options.  The  options  were  to  pay  the 

inventor  $50,000  for  the  6-month  option  or  not. 

1231  Only  two  options  were  available.  ,• 

1241  Adaptive.  The  options  were  not  a major  depar- 
ture from  ail's  normal  operations. 

1251  Static.  The  decision  to  be  made  could  not  be 

modified  later.  That  is,  this  decision  would  be 
made  final. 

13  Ease  of  Decision 

1312  Moderately  difficult.  The  correct  defcision  was, 
not  obvious  to  the  decision  maker  before  the 
analysis  was  done. 

1321  A very  familiar  type  oT  decision.  The  division 
president  made  these  kinds  of  decisions  very 
often. 
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1334  Choice  was  the  key  consideration;  it  was  nore 
important  than  inference  or  valuation. 

14  Stakes 

1421  Low-cost  swing.  The  most  expensive  alternative 
cost  $50,000,  and  the  le:;ast  expensive  cost 
nothing . 

1432  Before  the  analysis  was  made,  AIL  personnel 
judged  that  the  difference  between  the  best 
outcome,  winning  a large  contract,  and  the  worst 
one,  investing  in  promotion  and  development  and 
receiving  no  contract,  was  on  the  order  of 
several  million  dollars  in  present  valued  (PV) 
earnings. 

1442  Before  the  analysis,  the  expected  difference  in 
value  between  the  two  decision  options  wa^  of 
the  order  of  $100,000  in  PV  earnings. 

15  Outcome  Valuation 

1510  Easy  outcome  valuation.  When  the  uncertainties 
are  resolved,  earnings  and  return-on-capital 
(ROC)  would  be  readily  identifiable  for  this 
product,  and  these  two  summary  value  measures 
sufficiently  described  the  attractiveness  of  the 
outcome  to  the  decision  maker. 

1522  A low  number  of  value  dimensions  were  important. 
The  earnings  flows  and  returns  on  capital  (ROC) 
in  each  of  ten  years  were  important  to  the 
decision  maker. 
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1532  Exact  metric.  The  value  dimensions  were  easily 
measured  in  dollars  (for  earnings)  and  percent 
(for  ROC)  . 

1543  High  natural  combinability . Although  the  return 
on  capital  flows  neither  aggregate  naturally 
into  a single  figure  nor  combine  naturally  with 
the  earnings  flows,  the  earnings  flows  combine 
very  naturally  into  a single  present-value 
figure.  On  the  whole,  we  consider  this  to  be  a 
high  degree  of  natural  combinability. 

1552  Medium  time  horizon.  The  effects  of  the  deci- 
sion could  extend  over  10  years. 

1561  Overall,  the  component  valuation  was  very  easy. 
However,  the  return  on  capital  valuation  was  a 
bit  more  difficult  than  the  earnings  valuation. 

16  Outcome  Uncertainty 

1613  Many  uncertain  quantities  influenced  the  deci- 
sion. These,  included: 

A.  Would  the  option,  if  purchased,  be  exer- 
cised? 

B.  What  was  the  probability  of  receiving  a 
contract? 

1.  When  would  a legislative  or  FAA  man- 
date occur? 

a.  Would  a plane  crash  occur  in  the 
near  future? 
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b.  Would  the  mandate  contain  the 
needed  specific  standards? 

2.  Would  tough  competition  develop? 

C.  What  earnings  would  result  from  the 

contract? 

1.  How  many  units  could  be  sold? 

2.  What  price  could  be  obtained 
per  unit? 

3.  VJhat  profit  margin  would  result? 

4.  Would  AIL  win  a bid  for  the 
entire  contract  or  j'.ist  part 
of  it? 


1622  Medium  assessability . It  was  modcsrately  diffi- 
cult to  determine  the  appropriate  probability 
distributions  of  the  uncertain  quantities. 


1632  Medium  amount  of  uncertainty.  The  relative 
credible  Intervele,  901  Credible^ Interval,^ 

of  all  distributions  were  between  .10  and  1.00. 


1643  The  subsequent  act  of  whether  or  not  the  option 
would  be  exercised  critically  affected  the 
outcome . 


1653  The  uncertainties  in  the  case  lent  themselves 
naturally  to  indirect  judgment  techniques.  The 
probability  of  a contract  was  easily  thought  of 
in  terms  of  conditioning  events,  and  the  con- 
tract's earnings  were  easily  decomposed  into 
their  constituent  prxts  (see  1613  above) . 
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J661  Low  hindsight  monitoring.  The  total  outcome  of 
the  decision  could  not  be  known  for  about  10 
years. 

Decision  Processes 


21  Reaction  Time 

213  Days  of  Reaction  Time.  A commitment  of  funds — 
paying  a $50# 000  license  fee--was  required 
within  two  weeks  of  when  the  decision  analysis 
was  first  considered. 

22  Analytic  Processes 

2211  or  2212  At  most,  a total  of  two  or  three  sources 
were  available  to  provile  all  inputs  (values  and 
probabilities) . 

22212  and  22223  A group  of  decision  analysis 

consultants  were  readily  available  to  perform 
the  analysis. 

2231  It  was  ail's  convention  to  use  the  present  value 
of  pretax  earnings,  discounted  at  20%,  to 
evaluate  projects  like  this  one. 

2240  No  restrictions  constrained  the  reporting  pro- 
cedure. 

2252  "Trial  run"  analysis.  This  decision  was  chosen 
as  a trial  application  of  decision  analysis,  in 
part  to  help  determine  the  usefulness  of  deci- 
sion analysis  for  other  of  AlL's  decisions. 
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23  Organizational  Processes 


23111  This  analysis  was  initiated  spontaneously 

in  response  to  an  unanticipated  opportunity. 

23121  The  decision  was  initiated  by  AlL's  division 
president,  the  decision  maker. 

23211  The  user  of  decision  analysis,  the  division 

president,  is  also  the  person  who  will  make  the 
decision. 

23221  A small  group,  consisting  of  the  division 

president,  the  vice  presidents  for  operations 
and  business  development,  and  the  director  of 
long-range  planning,  shared  the  responsibility  of 
investigating  this  opportunity  and  making  a 
decision  on  it. 

2330  This  decision  did  not  need  to  coordinate  w'.th 
other  decisions. 

2340  The  division  president  did  not  ne^^d  to  justify 
his  decisions. 

2350  This  was  not  a controversial  decision. 

2361  Loose  performance  control.  The  division  presi- 
dent is  accountable  for  his  division's  results 
to  the  corporate  headquarters,  mainly  through 
periodic  reports  of  the  division's  profits. 

2372  The  rational  actor  model,  which  hypothesizes  a 
single  decision  maker  who  relies  on  reasoned 
judgments  in  making  his  decisions,  is  met  very 
well  in  this  case.  There  is  a single  decision 
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maker,  the  division  president,  who  will  make  his 
decision  baaed  on  a reasoned  judgment. 

2381  The  organization  was  slighcly  risk  averse. 

24  Decision  Maker  Characteristics 

24113  High-level  authority.  The  division  president 
was  the  chief  executive  of  the  division. 

24211  Little  familiarity  with  decision  analysis. 

The  decision  maker  had  some  familiarity  with 
decision  analysis  as  well  as  with  other  quan- 
titative methods. 

25  Resources  Available 

2512  A large  computer  facility  that  contained  special- 
purpose  decision  analysis  programs  (Harvard’s 
MANEJON  and  SRI’s  TREE  and  CTREE)  was  available 
in  a time-sharing  mode. 

2522  A strong  staff  was  available,  including  the 

director  of  long-range  planning,  who  was  familiar 
with  decision  analysis  and  with  the  substantive 
issues  of  this  decision  problem. 

2532  Decision  analysis  specialists  were  readily 
available . 

2541  Little  decision-maker  availability.  The  deci- 
sion maker  was  available  fcr  a half-day  briefing 
at  the  end  of  a two-week  analysis  period. 

25512  High  availability  of  assessors.  The 

probability  and  value  assessors  were  available 
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for  daily  involvements  with  the  analysis,  each 
involvement  lasting  several  hours. 

25522  The  staff  members  who  were  also  the  assessors 
were  familiar  with  decision  analysis. 


4.0  CASE  3:  A DESIGN-TO-COST  EVALUATION 


4 . 1  Introduction 


This  case  refers  to  a decision  analysis  performed  by 
Decisions  and  Designs,  Incorporated  for  the  Naval  Electronic 
Systems  Command  (NAVELEX)  to  evaluate  alternative  design 
proposals  for  an  Electronic  Warfare  (EW)  system.  A technical 
description  of  this  analysis  is  contained  in  Hays,  O'Connor, 
and  Peterson  (1975) , and  much  of  the  following  description  is 
excerpted  from  that  document. 

4.1.1  Problem  background;  The  design-to-eost  concept  - 
Faced  with  increasing  costs  and  tightening  budgets  in  the 
procurement  of  major  military  systems,  the  Department  of 
Defense  (DoD)  promulgated  a new  procurement  policy  in  1971 
linking  designed  performance  and  its  attendant  benefits  to 
cost  ceilings.  This  policy,  which  was  enunciated  in  DoD 
Directive  5000.1  and  subsequently  implemented  by  means  of 
instructions  published  by  each  of  the  Military  services, 
stated  that: 


discrete  cost  elements  (e.g.,  unit  production  costs, 
operating  and  support  cost)  shall  be  translated  into 
'design  to'  requirements.  System  development  shall  be 
continuously  evaluated  against  these  requirements  with 
the  same  rigor  as  that  applied  to  technical  requirements. 
Practical  tradeoffs  shall  be  made  between  system  capa- 
bility, cost,  and  schedule.  Traceability  of  estimates 
and  costing  factors,  including  those  for  economic  esca- 
lation, shall  be  maintained.^ 


^Acquisition  of  Major  Defense  Systems,  DoD  Directive  5000.1, 
July  13,  I'Sn. 
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The  intent  of  this  policy  statement  was  to  emphasize  the  need 
to  balance  (by  means  of  an  interactive  design  process)  mili** 
tary  costs  and  management  with  the  quantities  required  to 
achieve  the  most  effective  military  capability  with  available 
DoD  resources. 

The  design-to-cost  concept  introduced  by  DoD 
Directive  5000.1  has  recently  been  superceded  by  DoD  Directive 
5000. 28 1 which  further  states  that; 

The  Design  to  Cost  concept  establishes  cost  as  a design 

paraimeter  during  a system's  design  and  development  phase 

and  provides  a cost  discipline  to  be  used  throughout  the 

2 

acquisition  and  operation  of  a system. 

Changes  in  procurement  policies  obviously  involve 
new  procurement  practices  and  require  new  responses  in  system 
design.  Design-to-perf orraance  and  design-to-cost  procurement 
policies  both  relate  performance  and  cost,  but  each  does  so 
differently,  by  placing  different  emphasis  upon  one  or  the 
other.  Since  their  impact  upon  system  design  is  also  dif- 
ferent, an  understanding  of  the  differences  between  the  two 
kinds  of  procurement  policies  as  they  affect  system  design 
encompasses  the  reasons  for  the  change  and  suggests  means  of 
adapting  to  it. 

Under  a design- to-performance  procurement  policy, 
performance  requirements  in  the  RFP  are  stated  as  absolute 
descriptions  of  needs.  Any  system  failing  to  meet  the  stipu- 
lated requirements  is  defined  as  essentially  worthless.  If, 
for  example,  the  requirement  that  a military  system  respond 
within  ten  minutes  to  a distant  threat  were  taken  literally, 

2 

Design  to  Cost,  DoD  Directive  5000.28,  May  23,  1975. 


a system  that  responded  in  ten-and-one-half  minutes  would  be 
regarded  as  worthless.  However,  in  most  cases,  such  a system 
would  not  be  worthless,  only  worth  less  than  a system  actually 
satisfying  the  requirement.  Taken  literally,  such  a require- 
ment implies  a price-is-no-object  attitude  toward  design 
because  the  technology  necessary  to  satisfy  the  requirement 
may  be  exremely  expensive.  Although  seldom  reflected  in 
practice,  such  an  attitude  is  implicit  in  performance  require- 
ments and  can  lead  to  over-designed  systems.  Since  over- 
design  is  as  much  an  economic  as  an  engineering  concept,  a 
design-to-performance  procurement  policy  may  over-emphasize 
performance  and  lead  to  cost  overruns,  reduced  acquisition,  or 
compensatory  sacrifices  in  operational  performance  elsewhere 
in  the  system. 

Under  a design-to-cost  procurement  policy,  these 
problems  may  be  reduced  or  eliminated.  Rather  than  stating 
absolute  performance  requirements,  the  policy  stipulates  a 
production  cost  ceiling  for  each  unit  of  the  system  and  allows 
the  levels  of  performance  to  vary  within  certain  limits. 

These  variable  levels  of  performance  are  suitably  valued 
across  that  range.  Accordingly,  since  the  design  of  a system 
requires  the  maximum  value  of  performance  within  a budgetary 
constraint,  over-design  is  less  likely,  military  value  is 
maximized,  and  the  risks  of  cost  overruns,  insufficient 
acquisitions,  and  compensatory  sacrifices  in  operational 
performance  are  minimized  in  most  cases.  (Although  there  is 
some  controversy  whether  or  not  more  modest  levels  of  per- 
formance dictated  by  cosc-per-unit  ceilings  will  be  sufficient 
for  most  mission  requirements  in  certain  special  cases,  this 
is  not  an  issue  addressed  in  this  case.) 

For  a contractor,  the  most  immediate  implications 
involve  his  responsibilities  for  a new  system.  Previously, 
when  a Military  Service  defined  the  technical  requirements  of 
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a system,  the  contractor's  primary  responsibility,  theore- 
tically, was  to  propose  a system  design  meeting  those  require- 
ments. He  also  attempted  to  determine  a means  of  producing 
the  proposed  system  within  minimum  production  standards  of 
quality  at  a cost  lower  than  that  of  a competitor.  Presently, 
since  a Military  Service  simply  outlines  the  mission  require- 
ments of  a system  in  rather  general  terms  and  states  the 
production  cost-per-unit  ceiling,  and  since  all  contractors 
have  the  same  production  ceilings  and  standards  to  meet,  their 
primary  responsibility  is  to  propose  a system  design  offering 
more  and  efficient  technical  advantages,  not  lower  cost,  than 
does  a system  design  developed  by  a competitor. 

In  practice,  the  design-to-cost  procurement  policy 
is  governed  by  three  dominant  considerations:  the  probability 
of  meeting  production  and  installation  schedules,  the  accept- 
ability of  development  and  acquisition  costs,  and  military 
worth. 

First,  after  issuing  a Request  for  Proposal  (RFP) 
and  receiving  proposals  in  reply,  the  Military  Service  must 
estimate  the  likelihood  that  each  contractor  can  meet  the 
production  and  installation  schedules,  were  his  proposal 
accepted.  This  estimate  is  based  upon  assessments  of,  among 
other  considerations,  the  contractor's  past  perfoirmance  in 
fulfilling  government  contracts  and  his  capabilities  to 
produce  the  proposed  system  along  with  his  other,  possibly 
unrelated,  production  commitments. 

Second,  the  Military  Service  must  judge  the  ex- 
pense required  to  develop  a working  prototype  for  testing. 
Since  the  burden  of  designing  a system  under  design-to-cost 
procurement  policies  falls  largely  upon  each  contractor,  the 
Military  Service  has  paid  selected  contractors  to  develop 
their  proposals. 
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Third  (and  of  major  concern  in  this  case  study) , 
the  Military  Service  must  evaluate  the  military  worth  of  a 
contractor's  design  proposal  and  compare  that  proposal  with 
other,  often  quite  different,  proposals.  For  implicit  in 
design-to-cost  procurement  policies  is  a tacit  reliance  upon 
the  ingenuity  of  each  contractor  to  achieve  maximum  value  for 
a stipulated  cost.  The  flexibility  permitted  by  the  absence 
of  specific  technical  requirements  allows  each  contractor  to 
develop  a proposal  v.'hich  appears  to  him  to  best  meet  the 
requirements  of  a given  system.  Since  flexibility  allows 
diversity — each  contractor  develops  a design  that  reflects  his 
particular  corporate  interests,  his  understanding  of  the 
mission,  and  his  technical  and  manufacturing  capabilities-- 
some  method  of  evaluating  system  design  proposals  that  may  be 
quite  different  from  each  other  is  required. 

4.1.2  Decision  setting  - Under  this  gevieral  directive 
within  the  DoD  to  use  the  design- to-cost  procurement  policy, 
the  Naval  Electronics  Systems  Command  (NAVELEX)  was  faced  with 
a procurement  decision  for  an  Electronic  Warfare  (EW)  system, 
a group  of  sophisticated  and  complex  semi-automated,  com- 
puterized suites  for  detecting  and  responding  to  different 
threats  to  naval  surface  vessels  in  various  situations.  In 
particular,  NAVELEX  had  to  choose  to  fund  two  contractors  from 
among  six  who  responded  to  the  RFP  to  develop  their  systems 
from  the  design  stage  to  the  prototype  stage. 

The  decision  process  for  choosing  the  two  con- 
tractors was  fairly  complicated.  Ultimately,  the  decision 
would  be  made  by  NAVELEX' s source  selection  authority  (SSA) . 
His  decision,  however,  would  be  influenced  heavily  by  a recom- 
mendation that  he  would  receive  from  the  source  selection 
advisory  council  (SSAC)  that  was  formed  to  evaluate  the  pro- 
posals. The  SSAC,  in  turn,  would  base  its  recommendation  on 
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evaluations  of  the  proposals  that  it  would  receive  from  the 
source  selection  advisory  board  (SSAB) . 

At  the  point  when  decision  analysis  was  first 
considered  in  this  case,  the  project  manager,  who  headed  the 
SSAB,  was  faced  with  the  task  of  ensuring  that  the  SSAB  had 
its  evaluations  completed  within  six  weeks.  He  also  had  to 
ensure  that  the  evaluations  reflected  the  judgments  of  several 
EW  experts.  He  further  had  to  be  sure  that  his  evaluations 
could  be  justified  both  up  the  line  of  authority  to  the  SSAC 
and  the  SSA  and  to  the  contractors.  This  justification  to  the 
contractors  was  especially  important  because  a contractor  who 
was  justly  dissatisfied  with  his  system's  evaluation  could 
formally  protest  the  evaluation  and  thereby  slow  down  the 
whole  procurement  process. 

Although  the  design-to-cost  policies  have  been  in 
effect  since  1971,  their  previous  implementation  was  limited 
to  systems  already  beyond  the  design  and  development  stage  and 
ensured  only  that  the  system  would  be  produced  within  pre- 
viously budgeted  limits.  This  evaluation  of  the  Navy's  EV? 
System  was  thus  the  first  time  that  design- to-cost  procurement 
policies  had  been  implemented  from  the  very  initiation  of  a 
program  to  procure  a major  military  system.  It  also  marked 
the  first  time  that  decision-analytic  techniques  were  con- 
sidered for  evaluating  proposals  in  the  military  procurement 
process. 


Thus,  because  this  was  the  first  time  that  these 
methods  were  used,  the  contemplated  analysis  would,  in  part, 
determine  the  future  use  of  both  the  design-to-cost  policy  and 
decision  analysis  techniques  for  military  procurement  decisions. 


4.2  Performance  Determinants  of  a Good  Analysis 


This  case  required  thre-i  major  types  of  performance.  The 
analysis  had  to  communicate  the  reasoning  surrounding  the 
evaluations  (P321),^  effectively  disaggregate  the  valuation 
(P112) , and  promote  the  management  of  existing  expertise 
(P122) . In  addition,  the  analysis  had  to  be  completed  within 
six  weeks  time  (P21)  and  within  ’its  budget  (P22) . 

Effective  communication  was  very  important  in  this  case 
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because  the  evaluations  were  very  controversial  (S2352)  and 
required  a substantial  aimount  of  justification  (S2343) . The 
communication  of  reasoning  surrounding  the  evaluations  was 
particularly  important  because  a contractor  could  protest  an 
evaluation  that  he  considered  unfair,  and  this  protest  would 
add  greatly  to  the  costliness  of  the  EW  system  (P22) . 

A large  amount  of  data  concerning  the  valuation  of  the 
proposals  was  available  at  the  beginning  of  the  case.  Thus, 
a good  analysis  would  have  to  use  the  existing  data  effec- 
tively (P122) . Finally,  because  the  overall  valuation  was 
very  difficult  (S1512)  but  lent  itself  naturally  to  indirect 
judgment  (S1654) , an  important  feature  of  the  analysis  was  how 
well  it  disaggregated  the  problem  into  manageable  sub-problems 
(P112)  . 

4. 3 Analytic  Choices 

Some  of  the  analytic  options  that  we  recommend  for  this 
case  are  summarized  in  Table  4-1.  The  following  sections 
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References  that  begin  with  the  letter  "P"  are  performance 
measures,  which  are  explained  in  detail  in  Section  C of 
Volume  III. 

4 

References  that  begin  with  the  letter  "S"  are  situation 
characteristics,  which  are  explained  in  Section  4.5  below. 


61 


1 USER'S  OPTIONS 


4 


INPUT  SPECIFICATION 


11  USE  DECISION  ANALYSIS  AT  ALL? 
112  yes 

12  DOLLAR  AMOUNT  OF  DECISION 
ANALYSIS 

122  medium 

13  ROLE  OF  DECISION  ANALYSIS 

1312  public 
1342  display 
1351  communication 

14  ORGANIZATION 

1419  consultants  to  perform  the 
analysis 

1433  experts  provide  inputs 

15  RESOURCES 

1511  decision  anaiysis  specialists 

2 MODEL  APPROACH  OrTIONS 

21  COMPLEXITY  OF  ANALYSIS 

212  moderataly  complex 

24  BAuANCE OF  EFFORT 

2411  low  effort  on  modeling  options 
24211  iow  effort  OP  modeling 
uncertainty 

24223  high  effort  on  modeling  value 

3 INPUT  STRUCTURE 

31  UNCERTAINTY 

3110  mostly  certainty  equivalents 
3141  little  detail 

34  VALUE 

3212  medium  comprehensiveness 
3222  substantially  dec  rmpoted 
32312  single  utility  index 
3241 1 linear  value  function 


42  EVENTS 

421  scenarios 

43  VALUE  CRITERIA 

4319  mixture  of  units 
4322  "floating  zero"  base 

4r  ELICITATION  TECHNIQUE 

4512  direct  for  probabilities 

4532  direct  for  values 

5 OUTPUT 

51  SPECIFICATION 

5121  single  value  for  each  option 

6 MODEL  MANAGEMENT 

61533  many  iterations  on  both 
value  and  probability 
assignments 


Table  4-1 

MAJOR  ANALYTIC  CHOICES  IN  THE  DESIGN  TO-COST  CASE 


62 


explain  the  reasoning  behind  the  most  important  of  these 
choices . 

4.3.1  Choice  of  the  amount  of  analysis  - Section  3.1.2 
of  Volume  1 contains  an  explanation  of  the  guidelines  for 
determining  the  proper  amount  of  decision  analysis  in  terms 
of  situational  characteristics.  This  case  can  be  characterized 
along  these  situational  dimensions  as  follows. 

Important  factors  favoring  a large  eunount  of 
decision  analysis: 

o The  choice  in  this  case,  the  evaluation  of  the 

various  proposals,  was  difficult  (S1313).  No  pro- 
posed system  was  obviously  suoerior  or  inferior  to 
the  others. 

o The  key  determinant  of  a good  analvsis  was  how 
well  it  provided  the  RSAC  with  the  complicated 
evaluation  information  that  it  needed  in  order  to 
make  a recommendation  (S1332) . 

o The  stakes  involved  in  this  decision  were  fairly 
high  (S1443)  . A V’rong  decision  could  cost  the 
Navy  on  the  order  of  $10  million. 

o The  opinions  of  several  experts  were  important  for 
this  evaluation  (S2215) . 

o The  evaluations  required  extensive  iustif ication 
both  within  the  Navy  and  to  outside  contractors 
(S2342) . 

Important  factors  against  a large  amount  of 
decision  analysis: 
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o The  project  manager  was  unfamiliar  with  the  tech- 
niques of  decision  analysis  (S24210) . 

Other  factors  favoring  a large  decision  analysis; 

o This  was  a current  choice  (Sllll) , as  existing 
proposals  were  being  evaluated. 


o The  choice  was  expected  to  recur  (S1124) . A similar 
model  would  be  needed  at  a later  time  eo  evaluate 
the  two  systems  that  were  retained  for  prototype 
development. 

o The  evaluation  of  the  proposals  was  difficult  {S1512K 

o The  evaluations  were  required  in  six  weeks  (S214) . 

o The  "rational  actor"  model,  which  hypothesizes  a 
single  decision  maker  who  makes  his  decisions  on 
the  basis  of  reasoned  judgments,  was  a close  approxi- 
mation to  this  decision  making  process  (S2372) . 


Other  factors  against,  a large  analysis; 

o The  uncertainties  were  easy  to  assess  (S1623) . 

o A modest  budget  was  available  for  the  analysis 
{S2562)  . 


This  characterization  indicates  that  a large 
analysis  should  have  been  used  in  this  case.  Most  impor- 
tantly, the  stakes  involved  in  the  decision  were  large,  and  the 
evaluations  had  to  be  justified  to  a large  audience.  In  such 
a case,  a large  sensitivity  analysis  is  a very  valuable  part 
of  a good  analysis.  However,  a modest  budget  of  about  $20,000 
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was  available.  Our  recommendation  is  that  about  $?.0,000  more 
would  have  been  justified  in  this  case. 

4.3.2  Type  of  analysis  - The  following  major  choices 
were  made  mainly  by  the  analysts  during  the  course  of  the 
analytic  effort.  These  choices  were  in  the  areas  of  the 
overall  modeling  strategy,  the  modeling  of  value,  and  the 
modeling  of  uncertainty. 

Overall  modeling  strategy  - The  analysts  made  five 
major  choices  about  the  overall  strategy  of  the  modeling 
effort.  These  included  the  decision  to  perform  the  analysis 
from  the  viewpoint  of  a public  institution,  to  emphasize  the 
display  and  communication  aspects  of  the  aid,  to  devote  most 
of  the  effort  to  modeling  value,  to  perform  only  a modest 
amount  of  input  iteration,  and  to  rely  on  two  experts  to 
provide  most  of  the  model's  inputs. 

The  choice  to  build  the  decision  model  for  the 
U.S.  Navy's  benefit  (A1312)  rather  than  for  the  personal 
benefit  of  the  project  manager  was  made  for  basically  the  same 
reason  that  a similar  decision  was  made  in  the  Foreign  Policy 
Decision  Case  (Section  2.0  above).  Specifically,  the  project 
manager  was  motivated  to  evaluate  the  proposed  EW  systems  from 
the  point  of  view  of  the  systems'  values  to  the  Navy.  In  this 
case,  it  was  especially  important  for  the  analysis  to  serve  a 
public  function  because  the  evaluations  would  have  to  be 
justified  and  defended  to  the  EW  project's  SSAC  before  a 
decision  would  be  made  on  which  contractors  to  retain  (S2342) . 

The  model  was  used  to  display  and  communicate  the 
reasoning  surrounding  the  evaluations  because  the  evaluations 
required  so  much  justification  (A1342} . In  fact,  we  feel  that 
more  attention  should  have  been  paid  to  the  communication 
aspects  of  this  analysis  than  was  actually  done.  Specifically, 
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more  atten'-.ion  should  have  been  paid  to  recording  and  uis- 
playing  the  rationale  for  choosing  certain  analytic  techniques, 
such  as  multiplicative  combinations  of  values,  and  to  identi- 
fying and  highlighting  the  key  points  that  controlled  the 
evaluations.  If  the  analysts  had  paid  greater  attention  to 
these  details  of  communications,  the  analysis  would  have  been 
much  more  valuable  to  the  project  manager.  In  particular., 
including  more  rationale  would  have  spared  the  project  manager 
the  task  of  explaining  every  detail  of  the  analysis  to  the 
SSAC  and  would  have  allowed  him  to  focus  their  attention  on 
the  key  substantive  issues  rather  than  the  more  trivial  tech- 
nical ones.  The  point  is  that  a very  precise  model  is  of 
little  value  if  its  reasoning  cannot  be  communicated  effec- 
tively. Thus,  we  generally  recommend  that  an  evaluation  model 
such  as  the  one  developed  in  this  case  be  accompanied  by  a 
large  amount  of  explanation. 

Most  of  the  analytic  effort  was  devoted  to  modeling 
value  (A24223) , and  little  effort  was  spent  defining  the 
options  (A2411)  or  modeling  the  uncertainties  (A24211) . Since 
the  key  function  of  analysis  was  to  provide  the  SSAC  with 
evaluations  of  the  proposals  (S1332)  and  since  the  valuation 
was  difficult  (S1512)  and  needed  to  consider  many  complicated 
factors  (S1523) , it  was  natural  to  devote  a large  amount  of 
effort  to  modeling  the  valuation.  In  addition,  the  options 
were  clearly  defined  by  the  six  contractors'  proposals  (S1221)  , 
and  the  uncertainties  were  both  relatively  unimportant  and 
fairly  easy  to  assess  (S1623) . Thus,  little  of  the  analysis 
effort  was  required  in  order  to  model  the  uncertainties 
adequately.  In  general,  we  recommend  this  distribution  of 
effort,  with  the  main  emphasis  on  valuation,  when  the  main 
purpose  of  the  analysis  is  valuation. 

In  the  actual  analysis,  checks  were  made  on  the 
validity  of  the  value  assessment  using  two  methods.  The  first 
checks  were  made  by  comparing  the  model's  initial  structure 
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and  input  assessments  with  the  judgments  of  several  groups  of 
experts.  The  second  checks  were  made  by  using  the  model  to 
evaluate  existing  EW  systems  and  comparing  the  model's  valua- 
tions against  the  intuitive  valuations  of  experts  who  were 
very  familiar  with  the  existing  systems.  This  ensured  that 
the  model  behaved  in  accordance  with  the  generally  accepted 
assessments  of  the  existing  technology.  On  the  basis  of  these 
checks,  the  model  was  refined  to  improve  its  external  validity. 
These  kinds  of  checks  are  ones  that  we  generally  recommend. 

llrifferent  kinds  of  checks  are  also,  in  general, 
appropriate,  but  they  were  not  used  extensively  in  the  actual 
analysis.  These  \re  checks  of  the  model's  internal  validity. 
Such  checks  are  attained  by  varying  the  probabilities  of 
uncertain  events  (the  scenarios)  and  the  importance  weights  of 
valuation  components/  noting  the  resulting  changes  in  the 
valuation  of  the  different  systems.  Such  checks  both  identify 
the  critical  aspects  of  the  model  and  determine  the  robustness 
oi;  the  model.  In  deference  to  the  analysts  in  this  case,  such 
checks  of  internal  validity  are  difficult  and  very  time  con- 
suming unless  a computer  is  used  to  parform  the  required 
calculations.  In  this  case  a computer  was  not  readily  avail- 
able (S2510) . This  condition  suggests  that  in  cases  where  a 
.large  evaluation  effort  is  anticipated,  steps  should  be  taken 
early  in  the  analysis  effort  to  obtain  access  to  a computer 
facility  if  such  a facil,ity  is  not  otherwise  readily  avail- 
able.^ 


To  re-phrase  the  above  analytic  choice  in  the 
terms  of  the  taxonomy,  a modest  cimount  of  iteration  was 


5 . 

Since  the  time  that  this  analysis  was  performed,  Decisions 
and  Designs,  Inc.  has  developed  computer  programs  specifically 
designed  to  perform  these  types  of  sensitivity  checks  on  this 
type  of  model.  These  programs  can  be  used  on  the  IBM  5100 
portable  computer. 


67 


performed  on  the  value  inputs  to  the  model  (A61512)  but  a 
large  amount  of  iteration  for  both  the  value  inputs  and  the 
probability  inputs  is  recommended  (A61533) . 

The  final  choice  on  overall  modeling  strategy  was 
to  use  two  EW  experts  from  the  Naval  Research  Laboratories 
(NRL)  as  the  primary  sources  of  inputs  for  the  model.  This 
choice  was  made  because  these  experts  were  available  to 
devote  much  time  to  the  analysis  (S2552)  and  because  they  were 
familiar  with  the  other  people  who  could  provide  the  required 
information  in  other  areas  of  expertise. 

Modeling  of  value  - The  analysts  made  three  major 
choices  about  the  modeling  of  value.  These  choices  were  to 
seek  a value  measure  that  would  be  moderately  comprehensive/ 
to  decompose  the  value  measure  substantially,  and  to  utilize 
"swing  weights"  for  the  value  assessments. 

Although  three  main  categories  of  value  were 
important  for  the  evaluation  of  the  proposals  (military 
worth,  cost,  and  management),  only  one  of  these,  military 
worth,  was  very  difficult  to  evaluate.  Furthermore,  military 
worth  was  composed  of  eight  elements,  only  one  of  which,  the 
technical  system  utility  (TSU) , was  difficult  to  determine. 
That  is,  other  elements  of  military  worth,  such  as  the  degree 
of  modularity  of  a proposed  system,  could  be  determined 
readily  by  direct  observation.  However,  a proposal's  TSU 
could  not  be  determined  readily  nor  could  the  reasons  sur- 
rounding the  evaluations  be  communicated  easily.  Thus,  the 
analysts  would  be  most  effective  if  they  concentrated  on 
modeling  TSU  rather  than  on  trying  to  represent  all  value 
categories.  So,  we  would  recommend  a moderately  comprehensive 
value  measure  (A3212)  rather  than  a completely  comprehensive 
one  (A3213) . This  was  done  in  the  actual  analysis. 
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The  TSU  valuation  was  decomposed  extensively  into 
its  constituent  parts  (A3222)  to  enable  an  easier  assessment. 
Because  o£  the  decomposition*  each  assessment  considered  only 
a single  area  of  expertise.  This  choice  was  indicated  because 
the  valuation  of  TSU  was  very  difficult  to  make  (S1512) . An 
additional  important  choice  was  the  way  to  decompose  the  TSU. 

TSU  was  decomposed  by  functions  rather  than  by  components. 

That  is,  the  proposals  would  be  evaluated  on  how  well  they 
performed  functions  like  januning  and  deception  rather  than  on 
how  well  their  components,  such  as  antennas  and  receivers, 
were  designed. 

Although  a component  decomposition  combined  with 
a large  simulation  of  the  systems  would  have  provided  a 
somewhat  better  evaluation,  such  an  analysis  was  not  nearly  as 
efficient  as  the  functional  decomposition.  A decomposition  by 
components  would  have  resulted  in  a much  costlier  and  time 
consuming  analysis  because  it  would  have  required  a large 
simulation  model  of  the  interactions  of  a system's  components 
in  order  to  evaluate  it  (because  the  components  of  an  EW 
system  interact  heavily).  The  functional  decomposition  focused 
attention  directly  on  the  performance  of  the  system  and  thus 
offered  a much  cheaper  and  quicker  analysis. 

The  method  of  swing  weights,  similar  to  that 
explained  in  Section  2.3.2  (the  Foreign  Policy  Case)  above, 
was  used  to  combine  the  assessments  of  the  elements  of  TSU 
into  a single  index.  In  this  case,  however,  both  multipli- 
cative and  additive  combination  rules  were  used.  The  major 
reasons  for  using  swing  weights  were  the  same  in  this  case  as 
in  the  Foreign  Policy  Case.  Specifically,  this  method  pro- 
moted an  efficient  elicitation  of  values  and  easy  comiriunica- 
tion  of  the  evaluations.  The  multiplicative  combinations  were 
included  because  the  assessors  judged  that  some  low  scores  on 
some  value  elements  would  drive  down  the  overall  TSU,  regardless 
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of  the  scores  on  other  value  elements;  a multiplicative 
combination  exhibits  this  behavior.  (See  pp.  22-24  of  Hays/ 
O'Connor,  and  Peterson  [1975]  for  a more  detailed  description 
of  the  multiplicative  combination  procedure.) 

Model  of  uncertainty  - Three  main  analytic  choices 
were  indicated  for  the  model  of  uncertainty.  First,  scenarios 
were  used  to  describe  the  uncertain  environment  in  which  the 
EW  system  would  have  to  operate.  Second,  certainty  equiva- 
lents were  assessed  for  each  proposal's  uncertain  performance 
on  each  element  of  value.  Third,  a simulation  model  was  not 
used,  even  though  one  was  available. 

The  main  reason  for  using  scenarios  to  describe 
uncertain  events  (A421)  was  that  scenarios  were  issued  to  the 
contractors  to  be  used  in  their  design  effort.  Since  the 
design-to-cost  policy  did  not  allow  the  Navy  to  state  its 
needs  in  terms  of  strict  performance  requirements,  the  sce- 
narios provided  a way  for  it  to  specify  what  it  wanted  the  EW 
systems  to  do.  in  particular,  the  Navy  told  the  contractors 
to  propose  EW  systems  that  would  operate  in  eight  scenarios. 

In  addition,  the  use  of  scenarios  promoted  a logical  valuation. 
Since  valuation  was  difficult  (S1512) , methods  were  needed  to 
make  this  task  easier  for  the  assessors.  In  this  case,  value 
assessments  were  easier  if  the  assessors  visualized  the  dif- 
ferent proposed  systems  operating  in  different  scenarios. 

Thus,  the  use  of  scenarios  enabled  the  assessors  to  think 
concretely  about  the  way  that  the  systems  would  operate,  and 
this  enabled  the  assessors  to  assign  accurate  values. 

Certainty  equivalents  were  used  to  assess  the 
performance  of  each  proposed  system  along  each  element  of 
value  (A3110) . Originally,  the  analysts  considered  modeling 
the  uncertain  performances  of  the  proposed  EW  systems  as 
probability  distributions  (A3112) . However,  they  later  found 
that  such  a structure  would  be  too  time-consuming  to  implement 
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(F212)  and  that  the  experts  were  comfortable  in  making  direct 
judgments  of  certainty  equivalents.  Thus,  the  simpler  cer- 
tainty equivalent  method  was  indicated. 

Although  a model  was  available  to  simulate  the 
performance  of  the  proposed  EW  systems,  this  model  was  not 
used  (A5310) . A simulation  would  have  been  very  costly 
(P224)  and  time-consuming  (P215) , and  the  static  nature  of  the 
operation  of  an  EW  system  permitted  the  alternative  formu- 
lation of  the  value  method  described  above.  A simulation 
model  may  have  been  recommended  if  more  dynaumic  systems,  such 
as  weapon  systems,  were  being  evaluated.  A simulation  model 
can  represent  the  dynaimic  aspects  of  the  situation  (such  as 
reloading  and  reacting  to  an  enemy's  actions)  that  cannot  be 
represented  in  the  static  valuation  model  that  was  used. 

4.4  Conclusions 


The  taxonomic  framework  seems  to  serve  as  a useful  guide, 
in  this  case,  for  a decision  maker  to  decide  on  the  proper 
amount  of  decision  analysis.  However,  the  stakes  and  justi- 
fication needs  of  this  situation  were  sufficient  to  indicate  a 
large  analysis,  though  the  analysis  budget  was  ultimately 
sufficient  to  determine  the  size  of  the  actual  analysis. 

The  taxonomies  also  offer  a language  that  is  useful  in 
explaining  the  most  important  analytic  choices.  In  addition, 
the  taxonomies,  had  they  been  available,  may  have  improved  the 
actual  analysis.  In  particular  more  of  the  analysts'  atten- 
tions could  have  been  focused  on  the  display  and  communica- 
tion needs  of  the  situation.  On  the  other  hand,  the  taxonomies 
are  not  presently  developed  to  a level  of  detail  that  permits 
a discussion  of  all  major  analytic  choices.  In  particular, 
the  choices  to  use  external  validation  and  to  decompose  value 
by  function  rather  than  by  component  could  not  be  explained 
within  the  taxonomies  as  they  are  presently  developed. 
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4.5  Appendix;  Situation  Characterization 


This  appendix  presents  a characterization  of  the  Design- 
to-Cost  Case  along  the  dimensions  of  the  situation  taxonomy 
that  is  described  in  Section  A of  Volume  III.  A summary  of 
this  characterization  is  presented  in  Table  4-2. 

1 Decision  Substance 


11  Basic  Situation 

1111  Current  choice.  The  problem  was  to  make  an 
immediate  evaluation  of  the  six  contractors ' 
proposals. 

1123  Few  recurrences.  The  present  analysis  would 
evaluate  the  contractors  for  the  purpose  of 
selecting  two  of  the  six  to  fund  for  further 
development.  It  was  anticipated  that  a similar 
analysis  could  be  used  to  decide  between  the 
two  finalists  at  a later  time. 

12  Options 

1212  Narrow  options.  The  options  were  how  to 
evaluate  the  six  proposals  with  respect  to 
their  military  worth. 

1221  Clear  options.  The  options  were  clearly  de- 
fined to  evaluate  the  six  proposals. 

1232  Few  options.  Only  six  proposed  systems  were 
under  consideration 

1243  Adaptive.  The  task  of  evaluating  proposals  for 
military  systems  was  a familiar  one  for  the 
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2 fiECISION  PROCESS 

21  reaction  TIME 
214  months 

22  analytic  PROCESSES 


1 decision  SUBSTANCE 

11  basic  SITUATION 

1111  current  choice 

1123  some  expected  recurrence 
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12  OPTIONS 

1212  narrow 
1221  clear 
1232  few 
1243  adaptive 
1251  static 

13  EASE  OF  DECISION 
1313  difficult 

1321  tamiliar  to  decision  maxer 
1332  information  Is  the  key 

consideration 

14  STAKES 

1443  maximum  option  irripact 
about  $10  million 

16  OUTCOME  VALUATION 

1512  difficult  net  valuation 
1623  many  value  dimensions 
1531  approximate  metric 
1541  low  natural  combinability 
1653  long  time  horiaon 
1562  moderately  difficult  component 
valuation 

16  OUTCOME  UNCERTAINTY 

1613  many  uncertainties 
1623  easy  to  assess 
1632  medium  amount  of  uncertainty 
1654  lends  itself  to  both  direct  and 
indirect  judgement 
1661  low  hindsight  monitoring 


2216  many  input  sources 
222 12\  group  of  decision  analysis 
22223/  consultants 
2232  some  constraints  on  analytic 
method 

2240  no  constraints  on 
documentation 

2252  high  interest  in  tryittg  new 
analysis  tools 

23  ORGANIZATIONAL  PROCESSES 

231 12  scheduled  decision 
23123  initiated  by  a superior 
23214  to  provide  information 
23222  shared  responsibility 
2330  no  coordination  with  other 
decisions 

2342  pre-  and  post-decision  justification 

2352  very  controversial 

2372  ' tional  actor"  model 

24  decisionmaker 

CHARACTERISTICS 

24114  staff  member 
24210  unfamiliar  with  decision 
analysis 

25  RESOURCES  AVAILABLE 

2510  computer  not  readily  available 
2532  decision  analysis  specialist 

2542  decision  maker  available 

25512  assessors  available 
2562  modest  budget  available 
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SUMMARY  CHARAC 


Table  4-2 

TEHIZATION  OF  DESIGN-TO-COST  SITUATION 
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program  manager.  (The  method  of  evaluation, 
however,  was  new,  as  stated  in  1321  below.) 

1251  Static.  The  evaluations  made  would  be  final. 

Ease  of  Decision 

1313  Difficult  decision.  The  evaluations  of  the 
systems  were  not  at  all  obvious;  none  of  the 
six  contractors  appeared  either  totally  superior 
or  inferior  to  the  others. 

1321  Familiar  to  decision  maker.  The  project  manager 
was  familiar  with  evaluating  contractors' 
proposed  systems,  but  he  was  unfamiliar  with 
the  design-to-cost  procedure,  as  this  was  its 
first  application. 

1332  Information  is  key.  The  key  determinant  of  a 
good  analysis  was  how  well  it  provides  the 
evaluation  information  needed  by  the  SSAC  to 
make  its  recommendation. 

Stakes 

1443  Maximum  option  impact  of  about  $10  million. 

This  figure  is  an  estimate  that  a wrong  choice 
of  contractors  could  cost  about  10%  of  the 
total  proposed  system.  (The  10%  estimate 
considers  both  the  cost  due  to  a degraded 
system  and  the  cost  of  a possible  delay  due  to 
a justifiable  protest  by  one  of  the  contractors 
who  was  rejected.)  The  total  system  purchase 
would  be  approximately  30  units  costing  $1.2 
million  each,  50  units  costing  $500,000  each, 
and  120  units  costing  $300,000  each. 


15  Outcome  Valuation 


1512  Difficult  net  valuation.  Because  of  the 

design~to-cost  policy,  the  systems  could  not  be 
evaluated  on  the  basis  of  their  cost  alone. 
Evaluation  would  have  to  be  on  the  basis  of  the 
system's  performance  in  a number  of  different 
military  scenarios,  and  thus  the  evaluation  was 
difficult. 

1523  Many  value  dimensions.  The  operational  per- 
formance of  the  system's  many  parameters  had  to 
be  considered  in  many  operating  situations.  An 
example  of  one  of  the  hundreds  of  value  dimen- 
sions is  the  effectiveness  of  the  system's 
sensors  in  identifying  enemy  decoys  in  an  open- 
sea  battle  scenario. 

1531  Approximate  metric.  Most  of  the  system  para- 
meters could  be  characterized  in  metrics  such 
as  the  number  of  targets  that  the  system  could 
detect  before  saturation.  However,  these 
metrics  were  only  indirectly  related  to  the 
system's  military  value. 

1541  Low  combinability . The  metrics  for  the  many 
components  did  not  combine  naturally. 

1553  Long  time  horizon.  This  decision  would  help 
determine  the  Navy's  EW  capability,  and  it!r 
effects  would  extend  over  several  years. 

1562  Moderately  difficult  component  valuation. 

Since  the  options  were  developed  only  to  the 
stage  of  proposals,  it  was  not  easy  to  deter- 
mine with  accuracy  how  each  would  perform. 
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However*  each  valuation  c )uld  be  made,  with 
some  difficulty*  as  a judgment  of  the  certainty 
equivalent  of  a systt^m's  uncertain  performance. 

16  Outcome  Uncertainty 

1613  Many  uncertainties.  Not  only  was  each  system's 
performance  on  each  value  dimension  uncertain* 
but  situations  in  which  the  systems  would  have 
to  operate  were  also  uncertain. 

1623  Easy  to  assess.  Although  the  problem  contained 
many  uncertainties,  they  were  all  fairly  easy 
to  assess. 

1632  Medium  amount  of  uncertainty. 

1654  The  uncertainties  lent  themselves  to  both 

direct  and  indirect  judgment.  The  probabilities 
of  the  scenarios  could  be  assessed  directly* 
but  the  uncertain  performance  was  easiest  to 
assess  from  a decomposed  model  of  value. 

li.661  Low  hindsight  monitoring.  The  decibicn  maker 
could  never  expect  to  know  with  certainty 
whether  his  valuations  were  correct  (because 
the  rejected  systems  would  never  be  built) . 

2 Decision  Processes 

21  Reaction  Time 

214  Months.  Evaluations  of  the  systems  were  re- 
quired within  6 weeks  of  when  the  analysis  was 
first  considered. 
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.?2  Analytic  Processes 


2215  Many  input  sources.  Two  Navy  EW  experts  pro- 
vided most  of  the  assessments;  however,  they 
often  needed  to  obtain  their  information  from 
experts  in  other  substantive  areas. 

22212  and  22223  A group  of  decision  analysis 

consultants  were  readily  available  to  perform 
the  analysis. 

2232  The  analysis  had  to  be  performed  within  the 

guidelines  of  the  design-to-cost  policy  that  is 
explained  in  Section  4.1.1  above. 

2240  The  form  of  the  documentation  of  the  evalua- 
tions was  not  constrained. 

2252  Because  the  design-to-cost  policy  was  so  new, 

the  decision  maker  was  interested  in  trying  new 
analytic  tools  to  help  in  his  evaluations. 

23  Organizational  Processes 

23112  These  evaluations  were  known  about  and 

scheduled  around  the  time  that  the  RFP  was 
issued. 

23123  The  evaluation  was  initiated  by  the  SSA, 

who  would  eventually  decide  which  contractors 
to  retain. 

2314  The  results  of  the  project  manager's  evaluation 
would  serve  as  the  basis  for  the  SSAC's  recom- 
mendation for  the  SSA's  decision. 
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232222  snared  responsibility.  The  SAC,  SSAC, 

SSAB  and  several  other  experts  all  shared 
responsibility  for  various  aspects  of  this 
decision. 

f 

I 2330  This  decision  did  not  need  to  coordinate  with 

i others. 

I 2343  Pre-  and  post-decision  justification.  Any 

[ evaluation  that  the  SSAB  made  would  be  very 

controversial  and  thus  would  need  to  be  de- 
fended to  the  SSAC,  the  SSA,  and  to  the  bidding 

[ contractors.  The  evaluation  would  need  to  be 

i explained  to  the  SSAC  and  the  SSA  before  a 

decision  is  made,  and  to  the  contractors  after 
; a decision  is  made. 

f 

2352  Very  controversial.  This  dacision  evaluated 
six  strongly  competitive  contractors  and  was 

very  controversial. 

i 

2372  "Rational  actor"  model.  It  was  important  for 
the  evaluation  to  be  carefully  reasoned  and  to 
represent  the  view  of  the  U.S.  as  a whole. 

■ 24  Decision  Maker  Characteristics 

f 

f 

V 

I 24114  The  project  manager  was  a staff  member 

and  did  not  have  line  authority  for  deciding 
which  contractors  to  retain. 

24210  The  project  manager  was  unfamiliar  with 
decision  analysis. 

25  Resources  Available 
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2510  A computer  was  not  readily  available  to  the 
analysts. 

25^2  Decision  analysis  specialists  were  readily 
available  as  consultants. 

2542  and  25512  The  decision  maker  and  substantive 
experts  were  readily  available  throughout  the 
analysis.  At  least  one  person  was  available  on 
a full-time  basis  for  the  entire  6-week  period. 

2562  Modest  budget.  An  analysis  budget  of  about 
$20,000  was  available. 


I 


I 
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5.0  CASE  4:  A NAVY  TASK  FORCE  COMMANDER'S 
TACTICAL  DECISION  AID 


5.1  Introduction 


This  case  refers  to  a portion  of  a research  effort  per- 
formed for  the  Office  of  Naval  Research  to  develop  tactical 
decision  aids  for  Navy  task  force  commanders  (CTF's).  The 
case  below  is  presented  from  the  point  of  view  of  a CTF  who 
is  faced  with  a decision  problem  during  the  execution  phase 
of  a mission.  The  particular  aid  referred  to  here  is  the 
combined  parametric/Bayesian  model  that  was  first  identified 
in  Brown,  Peterson,  Shawcross,  and  Ulvila  (1975)  and  then 
developed  further  in  Peterson,  Randall,  Shawcross,  and 
Ulvila  (1976),  The  reader  should  refer  to  Brown,  et.al. 

(1975)  for  a technical  description  cf  the  aid  and  the  threat 
setting  for  which  it  was  developed. 

5.1.1  Problem  background  - A Navy  task  force  commander 
(CTF)  who  is  in  the  process  of  executing  a planned  mission 
may  be  faced  with  contingencies  that  would  force  him  to 
change  his  plan.  Some  such  contingencies  include: 

o severe  weather  conditions 

o a required  search  and  rescue  (SAR  incident) 
o an  unexpected  attack  by  a minor  power 

o an  unexpected  attack  by  a major  power 

A threat  that  concerns  a CTF  greatly  is  the  enemy  anti-ship 
missile  (ASM)  threat,  and  especially  the  threat  of  a surprise 
attack  from  an  aircraft/submarine  combination.  (In  this 
combination,  the  aircraft  serves  as  a targeting  platform  for 
the  submarine,  which  is  the  attack  platform  that  launches 
the  missile.) 
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Our  invastigation  of  this  threat,  which  is  re- 
ported in  Brown,  et.al.  (1975) , indicated  that  this  threat  is 
serious,  probable,  and  predictable.  The  threat  is  serious 
enough  to  justify  an  analysis  by  the  CTF;  it  jeopardizes  the 
survivability  of  the  task  force.  The  threat  is  probable 
enough  to  justify  a prior  analysis;  it  may  occur  across  a 
range  of  geographical  and  political  situations.  The  threat  is 
predictable  and  stable  enough  to  permit  a prior  analysis  that 
would  still  be  appropriate,  with  minor  adjustments,  when  the 
contingency  arose;  it  is  characterized  by  an  established 
sequence  of  observable  indicators. 

5.1.2  Use  of  decision  analysis  - At  this  time,  the  CTF's 
staff  does  not  possess  a decision  analysis  capability,  nor 
will  such  a capability  be  added  to  the  staff  within  the  near 
future.  Thus  any  aid  that  uses  decision  analysis  techniques 
must  be  designed  so  that  it  can  be  operated  by  people  who  are 
unfamiliar  with  its  technicalities.  However,  such  an  aid  may 
be  built  initially  by  decision  analysts  if  it  can  be  adjusted 
by  the  CTF  and  his  staff.  This  is  the  perspective  that  was 
taken  by  the  decision  analysis  researchers. 

5 . 2 Performance  Determinants  of  a Good  Analysis 

In  this  particular  case,  the  situation  dictated  that  a 
good  aid  was  one  that  performed  well  in  four  performance 
categories.  The  aid  had  to  operate  quickly,  provide  logical 
decisions,  perform  pre-choice  activities  well,  and  improve 
information  handling. 

Since  a CTF  would  have  only  a few  minutes  or  hours  to 
react  to  an  air/submarino  ASM  attack  (S211,  S212) the  most 


^Parenthetical  references  that  begin  with  the  letter  "S"  are 
to  the  situation  taxonomy  characterization  of  this  case, 
which  is  presented  in  Section  5.5  below. 
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important  performance  feature  of  an  aid  to  help  a CTF  in 

2 

this  situation  would  be  to  operate  very  quickly  (P2152) . 

The  aid  would  have  to  accept  inputs  quickly  (P2122) , perform 
calculations  quickly  (P2132) , and  provide  output  that  could 
be  interpreted  quickly  by  the  CTF  (P2142) . This  last  aspect, 
quick  interpretability , is  an  especially  important  consideration 
because  the  CTF  is  unfamiliar  with  decision  analysis  (S24210) . 
Notice  that  this  requirement  for  speed  concerns  the  operation 
of  the  aid,  not  its  construction,  which  does  not  have  to  be 
particularly  fast. 

The  overall  logic  of  the  choice  (P114)  is  very  important 
because  the  choice  of  responses  to  the  ASM  threat  is  difficult 
(S1313)  and  controversial  (S2352),  and  because  it  involves 
high  stakes  (S1444).  In  addition,  sound  predictions  (P113) 
are  an  important  component  of  logic  because  the  intent  of 
che  enemy  is  both  critical  to  the  decision  and  very  difficult 
to  assess  (S1621)  Also,  the  overall  logic  is  improved  by 
an  effective  disaggregation  of  the  decision  problem  (P112) 
because  several  people  have  responsibilities  for  different 
aspects  of  the  decision  (S2212)  . 

Because  this  situation  deals  with  a contingent  choice 
(1112)  where  the  actual  decision  will  need  to  be  made  quickly 
(S211)  by  a decision  maker  who  is  unfamiliar  with  decision 
analysis  (S24210)  and  who  will  have  no  decision  analyst 
available  (S2531) , activities  preceding  the  choice  processes 
(P31)  are  very  important.  With  this  category  of  activities, 
the  aid  should  provide  a good  pre-analysis  of  the  anticipated 
decision  (P314),  identify  when  a decision  is  required  (P312), 
and  monitor  the  environment  for  indications  that  a problem 
exists  (P311) . 


2 

Parenthetical  references  that  begin  with  the  letter  "P"  are 
to  the  performance  measure  taxonomy,  which  is  described  in 
Section  C of  Volume  III. 
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Finally,  because  a vast  amount  of  information,  much  of  it 
irrelevant,  will  be  available  to  the  CTF  when  he  has  to  make 
his  decision,  an  effective  aid  must  be  able  to  identify  and 
process  the  relevant  data  efficiently  (P331) . 

5.3  Analytic  Choices 

The  recommended  analytic  options  for  this  case  are  sum- 
marized in  Table  5-1.  The  following  sections  explain  the 
reasoning  behind  the  most  important  of  these  choices.  In  this 
case,  all  analytic  choices  were  made  by  the  analysts,  not  the 
decision  maker. 

5.3.1  Choice  of  the  amount  of  analysis  - Section  3.1.2 
of  Volume  I explains  guidelines  for  determining  the  proper 
amount  of  decision  analysis  in  terms  of  situational  character- 
istics. This  case  can  be  characterized  along  most  of  these 
situational  dimensions  as  follows. 

Important  factors  favoring  a large  amount  of 
decision  analysis: 

o The  decision  options  are  clearly  defined  (S1221) , 

The  composition  of  a naval  task  force  and  the  range 
of  its  operations  are  clearly  defined. 

o The  choice  is  difficult  (S1313) . The  task  force's 
response  to  this  ASM  threat  is  not  easy,  especially 
since  the  enemy  is  expected  to  try  to  conceal  his 
intent. 

o Choosing  the  best  course  of  action  is  the  key  deter- 
minant of  a good  decision  (S1334) . 
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USER'S  OPTIONS 


4  INPUT  SPECIFICATION 


11  USE  DECISION  ANALYSIS  AT  ALL? 

112  yes 

12  DOLLAR  AMOUNT 

123  high 

13  ROLE  OF  DECISION  ANALYSIS 

1311  private  aid 

1322  optional  decision 

1331  contingent  analysis 

1343  both  optimization  and  display 

1351  communication 

14  ORGANIZATION 

1412  three  decision  makers  and 
staff  provide  inputs 

15  RESOURCES 

1521  use  a computer 

2 MODEL  APPROACH  OPTIONS 

21  COMPLEXITY  OF  ANALYSIS 

211  simple 

24  BALANCE  OF  EFFORT 

2412  appreciable  option  definition 
24213  high  uncertainty  modeling 
24222  high  value  modeling 

3 INPUT  STRUCTURE 

31  UNCERTAINTY 

3112  model  with  probability 
distribution 

3141  low  level  of  detail 

32  VALUE 

3213  highly  comprehensive 
3221  modest  decomposition 
32312  single  abstract  irtdex 
32411  linear  value  function 


42  EVENTS 
4222  specific 

43  VALUE  CRITERIA 

4319  abstract  unit 
4322  "floating  zero"  base 

44  INDIRECT  ASSESSMENTS 

4421  Bayesian  updating 
4431  decomposed  assessment 

5 OUTPUT 

51  SPECIFICATION 

5111  preferred  decision 

5121  single  value  for  each  option 

52  DISPLAY  FORMAT 

5211  & 5221  computer  graphics 

6 MODEL  MANAGEMENT 

61  MODEL  DYNAMICS 
61533  high  input  iteration 

62  CONTINGENT  ANALYSIS  INPUT 
SEQUENCE 

621 1 enter  values  early 

5231  enter  likelihoods  early 

6242  enter  data  late 


Tables  1 

SUMMARY  OF  ANALYTIC  CHOICES 
FOR  A NAVY  TASK  FORCE  COMMANDER'S  DECISION  AID 
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The  stakes  are  very  high  (S1444) . A wrong  decision 
could  provoke  a major  confrontation  between  the  US 
and  the  enemy. 

Important  factor  aaainst  decision  analysis: 

The  decision  maker,  the  OTP,  is  unfamiliar  with 
decision  analysis  (S24210) , and  he  will  not  have  a 
decision  analyst  available  when  he  uses  the  aid 
(S2530) . 

Other  factors  in  favor  of  decision  analysis: 

Valuation  of  outcomes  is  difficult  (S1512) . The 
CTF  must  weigh  the  effects  of  his  decision  on  the 
success  of  his  mission,  the  damage  sustained  by 
his  forces,  and  the  international  political  oosition 
of  the  U.S. 

The  key  uncertainty,  the  intent  of  the  enemy,  is 
difficult  to  assess  (S1621) . 

A ship-beard  computer  is  readily  available  for  the 
CTF' s use  (S2512) . 

A large  analysis  budget  would  be  available  (R2563) . 

Other  factors  against  decision  analysis: 

The  choice  is  a contingent  one  (51112) , 

Less  than  one  expected  occurrence  (S1121) . It  is 
uncertain  whether  any  particular  CTF  will  ever 
face  this  enemy  ASM  threat. 
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o Very  short  reaction  time  (S211) . The  CTF  will 

ultimately  have  only  a tew  minutes  to  react  to  an 
ASM  attack. 

In  this  case,  more  factors  argue  for  a large 
analysis  than  argue  against  it.  Most  importantly,  the  stakes 
involved  in  the  decision  are  high  enough  to  offset  the  fact 
that  the  decision  is  a contingent  one  with  less  than  one 
expected  occurrence.  That  is,  even  though  the  probability 
that  a particular  CTF  will  ever  face  an  enemy  air/submarine 
ASM  threat  is  low,  the  consequences  of  a wrong  decision  are 
important^ enough  that  an  analysis  is  still  justified.  Fur- 
thermore, because  any  decision  aid  would  have  to  be  operated 
by  a CTF  who  is  untrained  in  decision  analysis  without  the 
assistance  of  a decision  analyst,  if  decision  analysis  is  used 
at  all,  a large  amount  must  be  used.  A large  analytic  effort 
is  required  to  adapt  decision  analysis  techniques  so  that  they 
can  be  used  by  people  who  are  unfamiliar  with  the  methodology. 
Overall,  then,  we  recommend  a large  amount  of  decision  analysis 
for  this  situation. 

5.3.2  Other  analytic  choices  - In  developing  this  aid, 
the  analysts  had  to  keep  in  mind  the  setting  in  which  the  aid 
would  be  used,  that  is,  aboard  a ship  by  a Navy  task  force 
commander.  Thus,  most  of  the  analytic  choices  were  made 
considering  ways  to  package  decision  analysis  techniques  into 
an  aid  that  could  be  used,  without  the  help  of  a decision 
analyst,  by  a person  untrained  in  decision  analysis.  Six 
major  analytic  choices  were  made  regarding  this  aid.  A 
choice  was  made  to  model  the  contingency  in  advance,  to  build 
a parametric  aid,  to  use  a Bayesian  hierarchical  inference 
structure,  to  display  the  results  in  a "probability  triangle," 
to  plan  for  a highly  interactive  operation  of  the  aid,  and  to 
use  the  " swing -weight"  method  of  modeling  value. 
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The  first  analytic  choice  was  to  model  the  enemy 
air/submarine  ASM  threat  contingency  in  advance.  That  is,  to 
develop  a model  of  this  contingency  well  in  advance  of  its 
occurrence  and  store  the  model  in  a computer,  to  be  recalled 
as  a need  arises.  This  advance  modeling  consisted  of  several 
parts.  The  CTF's  value  system  was  modeled  in  advance  (A6211) ; 
the  prior  probabilities  needed  for  the  analysis  were  modeled 
in  advance  (A6221) ; likelihoods,  which  expressed  the  diagnos- 
ticity  of  data  that  was  anticipated  to  arrive  as  the  threat 
developed,  were  stored  in  advance  (A6231) . Data,  which  sig- 
naled the  development  of  the  threat,  were  not  modeled  in 
advance  but  were  left  to  occur  as  the  threat  developed  (A6242) . 

These  choices  to  build  most  of  the  decision  model 
in  advance  were  made  because  the  aid  would  have  to  operate 
very  quickly  (P2111,  P2132,  r2152)  when  the  threat  actually 
occurred;  the  only  way  to  provide  the  required  speed  was  to 
perform  most  of  the  analysis  in  advance.  Characteristics  of 
the  threat  were  also  important  for  advanced  modeling.  In 
particular,  the  contingency  had  to  be  so  predictable  that  only 
minor  last-minute  adjustments  would  be  required  to  bring  the 
model  in  line  with  the  actual  contingency,  when  it  occurred. 

Our  evidence  indicated  that  an  enemy  air/submarine  ASM  attack 
v;ould  develop  according  to  a sequence  of  events  that  were 

3 

highly  stereotyped  and  thus  could  be  predicted  in  advance. 

In  addition,  the  range  of  the  CTF's  available  option  could 
also  be  predicted  in  advance.  This  eunount  of  predictability 
allowed  the  analysts  to  assxune  that  the  structure  of  the  model 
could  be  pre-programmed,  that  the  probable  indicators  of 
threat  and  their  diagnosticity  could  be  pre-programmed,  and 
that  the  CTF's  decision  options  and  value  structure  could  be 
anticipated  approximately  and  pre-programmed. 


In  the  actual  development  of  the  aid,  the  analysts  sought  out 
situations  that  appeared  to  exhibit  this  kind  of  predictability 
and  thus  supported  the  development  of  a pre-programmed  aid. 
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In  addition r storing  an  analysis  ahead  of  time 
permitted  the  CTF's  environment  to  be  monitored  for  character- 
istics that  indicated  that  a decision  was  needed  (P311, 

P312) . This  monitoring  was  possible  because  the  model  could 
be  used  to  process  data  and  alert  the  CTF  when  enough  evidence 
was  present  to  consider  changing  his  mission  plan.  Modeling 
likelihoods  in  advance  allows  the  aid  to  improve  the  handling 
of  information  (P331)  by  identifying  the  important  information 
30  that  it  can  be  given  priority  in  processing.  Building  the 
model  ahead  of  time  also  promotes  a good  pre-analysis  of  the 
threat  (P314)  because  it  allows  the  CTF  to  simulate  the 
threat  through  exercising  the  model. 

The  second  major  analytic  choice  was  to  use  a 
parametric  decision  model  that  processed  inputs  in  a manner  to 
produce  decision  thresholds  on  a single  paraimeter.  In  this 
case,  these  thresholds  were  stated  in  terms  of  the  enemy's 
intent  to  use  his  air/submarine  ASM  capability.  A parametric 
model  was  chosen  mainly  because  it  is  fast  and  especially 
because  that  output  is  stated  in  terms  of  a single  parameter 
that  can  be  grasped  quickly  by  the  decision  maker  (P2142) . 

The  decision  to  use  the  probability  distribution  of  the  enemy's 
intent  as  the  pareuneter  was  made  because  this  paraitieter 
appeared  to  capture  the  essence  of  the  threat  situation  in  a 
way  that  enhanced  its  predictability,  and  predictability  is 
the  key  feature  that  permits  a pre-modeling. 

The  third  major  analytic  choice  was  to  use  a 
simple  Bayesian  hierarchical  model  of  the  uncertain  quantity, 
the  enemy's  intent.  This  particular  model  was  composed  of  two 
analytic  components.  It  used  Bayes'  Theorem  to  process 
information  bearing  on  the  enemy's  intent  (A4421) , and  it 
decomposed  the  assessment  of  the  probability  of  intent  (A4431) 
in  a hierarchical  way.  This  analytic  choice  promotes  an 
effective  disaggregation  of  the  model  (P112)  by  allowing  the 
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opinions  of  various  experts  to  impact  on  the  probability 
assessment.  This  choice  was  further  indicated  because  the 
probability  assessment  lent  itself  naturally  to  this  decom- 
posed assessment  technique  (S1653)  so  such  a technique  could 
improve  the  predictive  power  of  the  aid  (P113) . 

The  fourth  major  analytic  choice  was  to  display 
the  results  in  a "probability  triangle"  using  a graphic 
computer  display  (A5211,  A5221) . A "probability  triangle" 

(see  pp.  3-15  through  3-19  of  Brown,  et.al.  [1975]  for  a 
technical  description)  can  display  graphically  probabilities, 
thresholds,  and  the  relationship  betvieen  them  for  an  uncertain 
quantity  that  can  be  characterized  by  three  states.  This 
display  was  chosen  because  it  is  a form  that  can  be  readily 
interpreted  by  the  decision  maker  (P2142) , allowing  him  to 
quickly  grasp  the  most  important  aspects  of  his  decision 
situation.  While  this  form  of  graphic  display  is  a highly 
desirable  form  of  output,  it  is  possible  only  in  situations 
where  the  essence  of  the  uncertainty  can  be  simplified  to 
consider  ]ust  three  possible  states  (A3141) . Fortunately,  in 
this  case  such  a simplification  was  possible  without  seriously 
over-simplifying  the  analysis.  This  model  v/as  programmed  on  a 
computer  because  that  is  the  only  way  that  the  inputs  could  be 
accepted  and  the  output  calculated  quickly  enough  (P2122, 
P2134)  to  be  useful  in  a developing  threat  situation.  Com- 
puter graphics  were  needed  in  order  to  display  the  "proba- 
bility triangle"  (A5211,  A5221) . 

The  fifth  analytic  choice  was  to  use  the  method  of 
"swing  weights"  to  model  the  CTF's  value  structure.  Similar 
to  the  method  used  in  the  Foreign  Policy  Case  (Section  2; 3.2 


4 

A research  effort  directed  at  seeking  similar  graphic  dis- 
plays for  events  that  contain  more  than  three  states  is 
reported  on  pages  22-26  and  91-115  of  Peterson,  et.al.  (1976). 
The  conclusions  of  this  effort  are,  at  this  time,  too  tenta- 
tive to  recommend  a method  for  more  than  three  states. 
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above),  the  valuation  was  defined  by  specifying,  on  each  of  ,! 

three  value  dimensions  (S1522) , the  relative  value  to  the  J 

CTF  of  each  action/event  combination  on  a O-to-100  scale  and  j 

then  aggregating  across  the  value  dimension  by  weighting  !( 

each  scale  to  obtain  a weighted  average  net  value  for  each 
action/event  combination.  This  method  was  chosen  mainly 
because  it  enables  a decision  maker  who  is  untrained  in 
decision  analysis  to  interpret  the  values  quickly  and 
accurately  and  because  it  is  a method  that  enables  fast 
sensitivity  analysis  (A61533)  (see  the  discussion  of  sensi- 
tivity analysis  in  the  following  paragraoh) . 

The  sixth  important  analytic  choice  was  to 
design  the  aid  in  a way  that  would  promote  sensitivity 
analyses  (A61533)  by  varying  the  model  inputs  and  seeing  the 
results  quickly.  Such  an  aid  would  allow  the  CTF  to  change 
the  stored  values  easily  and  see  the  effects  of  these  changes 
immediately.  A large  amount  of  sensitivity  analysis  promotes 
a good  pre-analysis  of  the  decision  situation  (P314)  by 
allowing  the  CTF  to  thoroughly  examine  the  effects  that 
change  inputs  have  on  the  decision  and  thus  gives  him  a feel 
for  the  robustness  of  his  decision  options.  By  allowing  for 
each  change  in  the  input  values,  this  aid  also  improves  the 
logic  (P144)  of  the  model  by  allowing  last-second  adiustments 
to  the  model  to  incorporate  unforeseen  events. 

5 . 4 Conclusions 

The  taxonomies  appear  to  be  very  useful  for  deteirmining 
the  proper  amount  and  type  of  decision  analysis  for  this  case. 

In  particular,  the  taxonomies  appear  to  provide  a good  indi- 
cation of  what  kinds  of  decision  analysis  techniques  are 

useful  for  a contingent  analysis  and  what  additional  situa-  i 

I 

tional  characteristics  are  required  to  justify  the  use  of  any  ; 

decision  analysis  in  the  contingent  choice  situation.  However, 
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in  this  case,  the  taxonomy  appears  to  be  more  useful  from  the 
decision  analyst's  standpoint  than  from  the  decision  maker's 
standpoint.  In  particular,  the  use  of  decision  analysis  at 
all  depended  critically  on  the  specific  analytic  techniques, 
and  these  techniques  can  probably  be  chosen  only  by  an  analyst. 

5. 5 Appendix;  Situation  Characterization 

This  appendix  presents  a characterization  of  this  case 
along  the  categories  of  the  situation  taxonomy  presented  in 
Section  A of  Volume  III.  This  characterization  is  summarized 
in  Table  5-2  and  explained  below. 

1 Decision  Substance 


11  Basic  Situation 

1112  Contingent  choice.  The  decision  facing  the  CTF 
is  what  actions  to  take  if  an  ASM  threat  begins 
to  develop. 

1121  Less  than  one  expected  occurrence.  While  this 
threat  could  present  itself  in  a range  of 
geographical  and  political  situations,  it  is 
not  certain  that  a particular  CTF  will  ever  be 
faced  with  the  threat. 

1131  Operating  act.  The  CTF  will  consider  actions 
to  reposture  his  forces  against  the  threat, 
including  launching  an  attack  against  the 
enemy  aircraft  or  submarine. 

12  Options 

1212  Narrow  options.  The  CTF  will  make  commitments 
of  the  forces  under  his  direct  control. 
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1 DECISION  SUBSTANCE 


2 DECISION  PROCESS 


11  BASIC  SITUATION 


1112 

contingent  choice 

1121 

less  than  1 expected  occurrence 

1131 

operating  act 

12 

OPTIONS 

1212 

narrow 

1221 

clear 

1232 

few 

1252 

dynam  ic 

13 

EASE  OF  DECISION 

1313 

difficult  choice 

1321 

familiar  to  the  decision 
maker 

1334 

choice  is  the  key  consideration 

14 

STAKES 

1444 

high  maximuiu  option 
impact 

15 

OUTCOME  VALUATION 

151 1 

moderately  difficult 

1522 

medium  number  of  value 
dimensions 

1530 

no  natural  metrics 

1541 

low  natural  combinability 

1562 

moderately  difficult  component 
valuation 

16 

OUTCOME  UNCERTAINTY 

1612 

few  important  uncertainties 

1621 

difficult  to  assess 

1653 

lends  itself  to  indiiect  techniques 

1663 

high  hindsight  monitoring 

21  REACTION  TIME 
211  minutes 

22  ANALYTIC  PROCESSES 
2212  few  sources  of  input 

23  ORGANIZATIONAL  PROCESSES 

23211  CTF  will  make  a final  decision 
23221  little  dispersion  of 
responsibility 

2342  post-decision  justification 
2352  very  controversial 

24  DECISIONMAKER 
CHARACTERISTICS 

24113  flas  officer 
24210  unfamiliar  with  decision 
analysis 

25  RESOURCES  AVAILABLE 

2512  computer  readily  available 

2522  strong  staff 

2530  no  decision  analyst 


Table  5-2 

CHARACTERIZATION  OF  NAVY  TASK  FORCE 
COMMANDER'S  TACTICAL  SITUATION 
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1221  Clear  options.  The  CTF  will  decide  among  the 
clearly  defined  actions  that  he  can  take  with 
his  forces. 

1232  A few  decision  options  must  be  considered. 

1252  Dynamic.  Some  of  the  actions  that  the  CTF 
might  take  could  be  modified  by  his  later 
actions.  For  instance,  he  could  take  an  in- 
creasingly hostile  stance  toward  the  threat;  he 
does  not  need  to  make  an  irrevocable  decision 
to  attack  immediately. 

13  Ease  of  Decision 

1313  Decisions  in  this  threat  situation  are  very 
difficult,  especially  since  the  enemy  can  be 
expected  to  try  to  conceal  his  actual  inten- 
tions. 

1321  A CTF  if  femiliar  with  the  choices  available  to 
him  to  counter  the  enemy  ASM  threat. 

1334  The  CTF's  primary  concern  is  to  choose  his  best 
option. 

14  Stakes 

1444  The  stakes  in  this  decision  are  very  high.  A 
wrong  decision  could  conceivably  provoke  a 
major  military  confrontation  between  the  US  and 
the  USSR. 

15  Outcome  Valuation 
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1511  Overall#  it  is  very  difficult  to  compare  the 
attractiveness  of  the  possible  outcomes. 

1522  A moderate  number  of  value  dimensions  are  im- 
portant. Those  that  are  most  immediately 
important  to  the  CTF  are:  1)  his  success  in 
completing  a primary  mission;  2)  the  daimage 
sustained  by  U.S.  forces  in  a confrontation 
with  the  enemy;  and  3)  the  political  reper- 
cussions of  his  actions. 

1530  No  natural  metrics  exist  on  any  of  the  three 
value  dimensions  mentioned  in  1522  above. 

1541  The  value  dimensions  do  not  combine  naturally. 

1562  The  outcomes  are  fairly  difficult  to  evaluate 
along  the  value  dimensions. 

16  Outcome  Uncertainty 

1612  A number  of  uncertainties  about  the  way  that 
the  ASM  threat  may  evolve  are  potentially 
important  for  the  CTF's  decision.  However, 
these  uncertainties  can  be  summarized  ade- 
quately by  the  uncertainty  about  the  enemy ' s 
intent  to  use  his  ASM  capability. 

1621  While  the  important  uncertainties  can  be 
summarized  nicely  in  terms  of  the  enemy ' s 
intent,  enemy  intent  is  a very  difficult 
uncertainty  to  assess. 

1653  The  assessment  of  enemy  intent  lends  itself  to 
the  use  of  indirect  elicitation  techniques 
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especially  because  an  ASM  attack  will  be 
accompanied  by  a sequence  of  intelligence 
reports  of  the  enemy's  activities. 

1663  High  degree  of  hindsight  monitoring.  The 

outcome  of  the  CTF's  decision  will  be  known  to 
him  soon  after  he  takes  an  action. 

2 Decision  Process 

21  Reaction  Time 

211  to  212  Minutes  to  hours.  The  CTF  ultimately 
has  only  a few  minutes  to  react  to  an  ASM 
attack,  but  he  may  have  several  hours  to  react 
to  the  early  signs  that  such  an  attack  might 
occur. 

22  Analytic  Processes 

2212  or  2214  Few  sources  of  inputs.  The  CTF 

(or  his  surrogate)  will  provide  value  inputs 
and  members  ox  his  staff  or  other  intelligence 
experts  will  provide  inputs  concerning  the 
uncertainties. 

23  Organizational  Processes 

23211  The  CTF  will  make  the  final  decision. 

23221  A small  group  of  people  will  have  respon- 
sibility for  parts  of  the  decision.  For 
excutiple,  intelligence  analysts  will  be  respon- 
sible for  providing  estimates  of  the  enemy's 
intent  and  the  CTF  will  be  responsible  for 
taking  an  action. 

i 

[ 

t 
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2342  The  CTF  may  have  to  justify  his  actions  after 
the  fact. 

2352  Any  decision  that  the  CTF  makes  concerning  his 
response  to  a Soviet  ASM  threat  will  be  con- 
troversial. 

24  Decision  Maker  Characteristics 

24113  A CTF  is  a flag  officer. 

24210  The  CTF  is  not  likely  to  be  familiar 
with  decision  analysis. 

25  Resources  Available 

2512  A ship-board  computer  would  be  readily  avail- 
able. 

2522  A CTF  has  a large  staff  readily  available. 

2530  No  member  of  a CTP's  staff  is  a decision 

analyst;  thus  no  decision  analyst  would  be 
available  while  the  aid  would  be  used.  How- 
ever, a decision  analyst  would  be  available  to 
help  develop  the  aid. 
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6.0  CASE  5:  PREDICTING  NATO'S  RESPONSE 


6.1  Introduction 


This  case  refers  to  a study  performed  for  the  Studies. 
Analysis,  and  Gaming  Agency  of  the  Office  of  the  Joint  Chiefs 
of  Staff  and  reported  in  Brown,  Kelly,  Stewart,  and  Ulvila 
(1977)  and  Brown,  Kelly,  Stewart  and  Ulvila  (1975/2) . The 
study  was  an  attempt  to  predict  the  behavior  of  NATO  in 
response  to  the  question,  "If  the  Warsaw  Pact  countries  were 
planning  to  attack  at  the  end  of  a 30-day  mobilization  cycle, 
at  what  point  in  time  after  the  pact  began  to  mobilize  would 
NATO  initiate  a state  of  reinforced  alert?"  Since  the  main 
goal  of  the  study  was  to  develop  a model  for  predicting  the 
behavior  of  NATO  rather  than  to  analyze  a decision  problem, 
only  a fraction  of  the  analytic  choices  were  relevant.  For 
this  reason,  the  discussion  of  the  case  below  is  much  shorter 
than  that  of  the  other  cases  in  this  report. 

6.1.1  Problem  setting  - Early  in  1974,  the  Studies, 
Analysis,  and  Gciming  Agency  (SAGA)  of  the  Office  of  the  Joint 
Chiefs  of  Staff  was  interested  in  studying  the  effects  that 
alternative  Multiple  and  Balanced  Force  Reduction  (MBPR) 
actions  would  have  on  NATO's  capability  to  respond  to  an 
attack  by  the  Warsaw  Pact  countries.  Within  this  broad  topic 
area,  SAGA  was  most  interested  in  developing  a method  of 
predicting  when  NATO  forces  would  move  to  a state  of  rein- 
forced alert  if  the  Pact  was  planning  to  attack  after  a par- 
ticular 30-day  mobilization  scenario. 

6.1.2  Use  of  decision  analysis  - The  study  approached 
the  problem  of  predicting  NATO's  reaction  to  a Pact  threat  by 
considering  the  total  lag  in  NATO's  response  to  be  composed  of 
two  parts.  The  first  part  was  the  lag  in  time  between  the 
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Pact's  mobilization  and  a recommendation  by  the  Supreme  Allied 
Commander  in  Europe  (SACEUR)  to  the  North  Atlantic  Council 
(NAC)  for  NATO  to  mobilize.  The  second  part  was  a lag  between 
the  receipt  of  this  recommendation  and  a decision  by  the  NAC 
to  move  the  NATO  forces  to  a state  of  reinforced  alert. 

Within  this  framework,  the  major  analytical 
effort  was  devoted  to  developing  a prescriptive  model  of 
SACEUR' s behavior.  This  effort  took  the  form  of  a "rational 
choice"  model  to  predict  when  in  the  Pact's  mobilization  cycle 
SACEUR  would  prefer  to  see  NATO  forces  mobilize. 

Initially,  the  modeling  effort  made  strict  assump- 
tions about  the  mobilization  scenario,  SACEUR' s behavior,  and 
NAC's  behavior.  Later  in  the  analysis,  some  of  these  assump- 
tions were  relaxed  informally. 

6. 2 Analytic  Choices 

6.2.1  Amount  of  decision  analysis  - Since  the  main 
purpose  of  this  study  was  a prediction  rather  than  a decision, 
many  of  the  situation  characterizations  favoring  decision 
analysis  that  are  presented  in  Section  3.1.2  of  Volume  I are 
irrelevant  to  this  case.  Most  importantly,  the  characteriza- 
tions that  are  concerned  with  the  decision  options  and  with 
validation,  cannot  be  established  meaningfully  in  this  case. 
However,  other  characterizations  are  relevant.  With  respect 
to  these  characterizations,  this  case  can  be  classified  as 
follows . 

Situations  characteristics  strongly  favoring 
decision  analysis: 

o The  analysis  was  concerned  mainly  with  inference 
(S1333) . 
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o Several  sources  of  uncertainty  inputs  needed  to  be 
considered  (S2214) . 

Situation  characteristic  strongly  against  decision 

analysis : 

o Choice  was  not  a consideration  (S1334) . 

Additional  characteristic  favoring  decision 

analysis : 

o Uncertainties  were  difficult  to  assess  (S1621) . 

Additional  characteristic  against  decision  analysis: 

o The  "rational  actor"  model  was  a poor  approximation 
of  the  decision-making  process  (S2371) . 

Since  so  few  situation  characteristics  were  rele- 
vant in  this  case,  the  taxonomy  does  not  serve  a very  useful 
purpose  in  deciding  how  much  analysis  is  justified.  Our 
recommendation  in  this  case  is  that  a moderate  amount  of 
analysis  is  appropriate,  mainly  because  decision  analysis 
techniques  can  be  very  useful  in  making  difficult,  important 
inferences. 

6.2.2  Other  analytic  choices  - The  dei'ision  analysts 
made  five  major  analytic  choices  in  this  case.  First,  they 
chose  to  devote  the  major  portion  of  their  effort  to  a pre- 
scriptive model  of  when  NATO  would  mobilize.  Second,  they 
chose  to  use  scenarios  to  represent  uncertainties.  Third, 
they  chose  to  start  with  a fairly  simple  approximate  analysis. 
Fourth,  probabilities  were  elicited  directly.  Fifth,  the 
assessments  of  individual  experts  were  pooled  to  obtain  the 
final  probability  estimates. 
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As  mentioned  in  the  introduction,  most  of  the 
analytical  effort  in  this  case  was  devoted  to  developing  a 
•'rational  choice"  model  of  SACEUR's  decision  process.  This 
model  was  a prescriptive  decision  analysis  model  that  attempted 
to  identify  when,  in  a Warsaw  Pact  30-day  mobilization  and 
attack  cycle,  SACEUR  would  prefer  to  see  NATO  forces  move  to  a 
state  of  reinforced  alert.  The  critical  feature  of  this  model 
was  the  treatment  of  the  action  "NATO  mobilize."  The  model 
treated  this  as  an  action  in  the  classical  form  of  preposterior 
analysis  (A3131) . That  is,  for  each  day  in  the  Pact  30-day 
attack  cycle,  the  model  analyzed  whether  the  action  "NATO 
mobilize  today"  was  preferable  to  the  action  "NATO  wait,"  and 
the  day  that  mobilization  was  first  preferred  determined  the 
SACEUR  lag.  Such  a model  is  appropriate  when  the  "rational 
actor"  model  is  a close  approximation  of  the  decision-making 
process  (S2372)  and  when  the  structure  of  all  uncertainties 
can  be  predicted  accurately.  However,  these  conditions  were 
not  met  in  this  case,  and  this  type  of  model  should  not  have 
been  structured  as  an  inference  model  to  assess  directly  the 
likelihood  that  NATO  would  mobilize.  In  particular,  a direct 
group  assessment  of  the  probability  of  NATO's  actions  would 
have  provided  a better,  less  expensive,  and  less  complex 
prediction  model  than  the  "rational  actor"  did. 

A scenario  CA421) , a specific  description  of  the 
information  that  NATO  might  receive  if  the  Pact  were  mobilizing 
on  a 30-day  attack  cycle,  was'used  to  describe  the  event  space 
of  the  Pact's  actions  for  two  reasons.  First,  the  event  space 
was  very  complex;  the  Pact  countries  could  take  a wide  range 
of  political  and  military  actions  in  preparation  for  an  attack 
on  NATO.  Second,  it  was  impossible  for  the  analysts  to  obtain 
a convenient  summary  of  the  Pact's  actions  for  the  purpose  of 
predicting  NATO's  reactions.  In  such  a situation,  a scenario 
is  often  the  best  way  to  characterize  the  event  of  a Pact 
attack  because  the  scenario  provides  a concrete  description  of 
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events  that  enables  assessors  to  provide  accurate  probability 
and  value  judgments. 

A very  simple  {A211)  model  that  used  many  approxi- 
mations (A231)  was  indicated  for  similar  reasons.  A simple 
model  provided  the  best  cost  effectiveness;  the  improvement  in 
logic  that  a more  complex  analysis  would  provide  could  not 
justify  its  greater  cost,  especially  within  the  range  of  the 
analysis  budget  of  $60,000.  An  approximate  analysis  that 
contained  a very  simplified  tree  structure  and  relied  heavily 
on  the  use  of  certainty  equivalents  (A3110)  to  aggregate 
uncertainties  was  indicated  mainly  because  the  assessors  were 
not  readily  available  during  the  analysis  (S25511) . In  addi- 
tion, the  analysts  felt  that,  in  this  situation,  an  approxi- 
mate analysis  could  save  a great  deal  of  cost  at  little  expense 
in  terms  of  logic. 

The  probability  assessors  were  accustomed  to 
making  direct  assessments  of  probabilities  based  on  intelli- 
gence information.  Therefore,  the  analysts  could  make  the 
best  use  of  the  assessor's  time,  which  was  scarce  (25511),  by 
eliciting  the  probabilities  directly  rather  than  indirectly 
(e.g.,  using  such  techniques  as  Bayes'  Theorem  (A4421) . 

Finally,  the  analysts  asked  each  expert  for  his 
assessments  individually  and  then  the  analysts  averaged  the 
individual  assessments  to  estimate  the  group's  opinion  (A4540) . 
This  method  contrasts  with  the  alternative  of  having  the 
experts  resolve  their  differences  to  arrive  at  a group  assessment 
(A4541)  (e.g.,  using  the  Delphi  Technique).  Such  a consensus 
generally  requires  the  assessors  to  be  much  more  available 
than  they  were  in  this  case.  In  addition,  techniques  used  to 
elicit  a consensus  generally  are  more  costly  and  time-consuming 
than  the  averaging  technique.  However,  the  averaging  tech- 
nique generally  does  not  provide  inputs  that  are  as  sound  as 
those  provided  by  a group  assessment.  For  this  reason,  the 
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logical  consistency  of  the  inputs  was  checked  by  pooling 
several  models  (A6121)  of  the  same  inputs.  In  this  case,  the 
prediction  of  the  SACEUR  lag  that  resulted  from  using  the 
average  of  the  probability  assessments  was  in  general  agree- 
ment with  a direct  assessment  of  the  SACEUR  lag.  The  combined 
procedure  of  averaging  the  assessments  and  checking  the  results 
with  a pooling  of  models  provided  an  analysis  that  was  much 
cheaper  and  nearly  as  good  as  would  have  been  provided  using  a 
group -assessment  technique. 

6.3  Conclusions 


The  taxonomic  freunework  appears  to  be  less  well  suited 
for  indicating  analytic  choices  for  this  case  than  in  the 
previous  cases.  Because  the  case  was  interested  only  in 
making  inferences  about  NATO's  behavior,  most  of  the  situation 
categories  proved  irrelevant,  and  classifications  along  these 
categories  were  impossible.  Thus,  the  taxonomy’s  guidance  on 
the  amount  of  analysis  to  perform,  which  required  classi- 
fication along  some  of  the  irrelevant  categories,  was  of  little 
use  in  determining  the  proper  amount  of  analysis.  However, 
the  taxonomies  were  useful  for  explaining  the  reasons  sur- 
rounding the  most  important  choices  of  analytic  techniques. 

i 

i 

6.4  Appendix;  Partial  Situation  Characterization 

Since  the  purpose  of  this  analysis  was  to  develop  a model 
that  could  be  used  to  predict  NATO's  action  rather  than  to 
[ make  any  decision,  only  some  of  the  situation  characterizations 

[ were  relevant  in  this  case.  For  instance,  all  characterizations 

[ concerned  with  the  options  or  the  valuation  wore  completely 

irrelevant.  The  following  table.  Table  6-1,  and  discussion 
include  only  those  elements  of  the  situation  that  we  consider 
to  be  important  in  this  case. 
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1 DECISION  SUBSTANCE 


13  EASE  OP  DECISION 

1333  inference  is  the  key 
consideration 

16  OUTCOME  UNCERTAINTY 

1613  many 
1621  low  assessabiiity 
1633  high  amount  of  uncertainty 
1654  lends  itself  to  both  direct  and 
indirect  judgement 
1661  low  hindsight  monitoring 

2 DECISION  PROCESS 

21  REACTION  TIME 

214  months 

23  ORGANIZATIONAL  PROCESSES 

2371  "rational  actor”  model  is  a poor 
approximation 

24  DECISIONMAKER 
CHARACTERISTICS 

2421 1 unfamiliar  with  decision  analysis 

25  RESOURCES  AVAILABLE 

2551 1 low  availability  of  probability 
assessors 

25521  low  sophistication  of  probability 
assessors 


Table  6-1 

PARTIAL  CHARACTERIZATION  OF  THE  NATO  RESPONSE  SITUATION 
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Decision  Substance 


13  Ease  of  Decision 

1333  Inference  was  the  key  consideration  in  this 

situation.  SAGA  was  interested  in  predicting 
NATO's  behavior. 

16  Outcome  Uncertainty 

1613  Many  uncertainties.  Uncertainties  that  poten- 
tially impacted  on  the  prediction  of  NATO's 
action  included: 

o Actions  of  Warsaw  Pact  countries  . 
o Information  that  NATO  would  receive  about 
the  Pact's  actions. 

o SACEUR's  actions  in  response  to  the  infor- 
mation . 

o NAC's  actions  in  response  to  the  informa- 
tion. 

From  these,  several  more  specific  ones  were 
included  explicitly  in  the  model: 

1.  The  probability  that  the  Pact  would  attack 
given  a particular  scenario. 

2.  The  length  of  the  Pact's  attack  cycle. 

3.  The  length  of  SACEUR's  recommendation  lag. 

4.  The  length  of  NAC's  action  lag. 

1621  The  probabilities  that  were  modeled  were  diffi- 
cult to  assess. 
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1633  The  uncertain  quantities  were  very  uncertain. 

1654  Some  of  the  uncertainties  lent  themselves  to 

indirect  assessment  techniques  and  others  lent 
themselves  to  direct  ones. 

1661  Low  hindsight  monitoring.  It  would  be  very 
difficult  for  SAGA  to  monitor  the  accuracy  of 
their  prediction  of  NATO's  behavior. 

2 Decision  Process 

21  Reaction  Time 

214  Several  weeks  were  available  to  perform  the 
analysis. 

23  Organizational  Processes 

2371  The  "rational  actor"  model,  which  hypothesizes 

a unitary  decision  maker  who  makes  his  decisions 
based  on  reasoned  judgment,  does  not  capture 
the  essence  of  NATO's  complex  decision  making 
process  accurately. 

24  Decision  Maker  Characteristics 

24211  The  SAGA  analysts,  who  would  use  the  decision 
analysis  model,  were  unfamiliar  with  decision 
analysis  techniques. 

25  Resources  Available 

25513.  The  experts  who  would  supply  probability 

estimates  were  not  very  available  during  the 
analysis  effort. 
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25521  The  experts  who  would  supply  probability 

estimates  were  not  very  sophisticated  in  their 
knowledge  of  decision  analysis  techniques. 
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